EvilTwins.cs

When promoting elements to a stream, the error message "...name already exists in backing stream..." appears. What does this mean?

An element ID (EID) is an immutable unique identifier (number) assigned to the element when it is added to AccuRev. It allows AccuRev to keep track of the entire history of the element from the time it was first added to the depot should it undergo a name or location change, or made defunct. When two elements with the same name and location are added to the depot in disparate streams, when promoted up the stream hierarchy they will eventually collide. This situation is referred to as "evil twins". The error message indicates there are two elements with the same name and location but with different EID's. To fix evil twins:

1. Since you must clear the namespace in one of the streams so the "good twin" prevails, you must decide which twin you want to keep. Use AccuRev's Version Browser to view the history of each twin as this could impact your decision. For example, if one of the elements has gone to production and the other has not, you would want to make the production version the good twin and the other the evil twin.
2. If the element is a file, do a diff on the twins to make sure the good twin (the one that will prevail) has the correct content. Edit as needed.
3. Rename (move) the evil element and then defunct it. Note that as a best practice, renaming should always be done before making an element defunct, even in cases that are not evil twins (see Note below). In the case of evil twins, rename the evil twin with the ".EVIL" extension. Otherwise add a descriptive extension to the element, for example ".OBSOLETE".
4. Promote the renamed-defunct evil twin. Then do one of the following steps, based on a or b:
   1. If the good twin exists higher up in the stream's hierarchy:
      1. When the bad twin was renamed, the good twin was inherited immediately and appeared in your workspace as (missing). Right-click on the missing element, select Populate and select Ok to bring it into your workspace.
   2. If the good twin does not exist higher up in the stream hierarchy:
      1. Use the Change Palette to cross promote the good twin to your workspace.

Note

You CAN promote an element into a stream with the same name as another element as long as the element in the stream has (defunct)(member) status. In this case, the twin that is (defunct)(member) is made (stranded) thus allowing the promoted element with the same name to become the new active member in the stream. However, this can cause promotion problems later if a twin with the (stranded) element's EID exists higher up in the stream hierarchy. This would require promoting the (stranded) version before its twin can be promoted. The practice of renaming (move) elements before their defunct-promote operation helps alleviate this problem and is the reason for the recommendation in step 3 above.

Diff Against... File on Disk is the only way to diff files that are evil twins. For example diff -v Stream1\1 -V Stream2\1 \.\Dir\foo.java fails since internally it's implemented using the name -e option and does not rely on depot-relative pathnames. This allows diff to operate on files that have undergone namespace changes but (sadly) fails in the case of evil twins. AccuRev defect 26192, RFE 26215, Salesforce 00044449.

To locate:

EvilTwins.exe scans for the presence of evil twins in depots listed in EvilTwins.exe.config. Add depot entries (case-sensitive) as needed. Program output is sent to EvilTwinsFound-YYYY-MM-DD.log, a daily log file created/updated in the same folder where EvilTwins.exe resides. The program's return value is zero (0) if it ran successfully or one (1) in the event of an exception or initialization failure. Program errors are sent to STDOUT and logged in %LOCALAPPDATA%\AcTools\Logs\EvilTwins-YYYY-MM-DD.log.
Optional: To handle erroneously reported evil twins (false positives) or to disregard those that will eventually resolve on their own, put them in the TwinsExcludeFile specified in EvilTwins.exe.config to keep them from being reported in the future. Empty lines are allowed in the exclude file and comments can be added by using the # sign as the first character of each comment line. The full path to the exclude file should be given as the value for key TwinsExcludeFile in EvilTwins.exe.config. (Do not change the key name TwinsExcludeFile to some other name, otherwise the program will malfunction.)

EvilTwinsFound log file

\.\Mobile\LogManager.h
  EID: 4872 on Wed May 24 9:28 AM
    NEPTUNE_DEV2 (backed)
      Size: 411, ModTime: 8/3/2016 6:29:57 PM {text}
      Real: NEPTUNE_DEV2_thomas\5, Virtual: NEPTUNE_MAINT2\5
    NEPTUNE_MAINT1 (backed)
      Size: 407, ModTime: 7/5/2016 10:32:41 AM {text}
      Real: NEPTUNE_MAINT1_barnyrd\3, Virtual: NEPTUNE_MAINT\4
    NEPTUNE_MAINT2 (member)
      Size: 411, ModTime: 8/3/2016 6:29:57 PM {text}
      Real: NEPTUNE_DEV2_thomas\5, Virtual: NEPTUNE_MAINT2\5

  EID: 4989 on Wed May 24 9:28 AM
    NEPTUNE (member)
      Size: 412, ModTime: 8/3/2016 5:41:36 PM {text}
      Real: NEPTUNE_UAT_robert\1, Virtual: NEPTUNE\3
    NEPTUNE_CC (backed)
      Size: 412, ModTime: 8/3/2016 5:41:36 PM {text}
      Real: NEPTUNE_UAT_robert\1, Virtual: NEPTUNE\3
...

EvilTwins.exe.config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="Depots" type="AcUtils.DepotsSection, AcUtils, Version=1.6.4.0, Culture=neutral, PublicKeyToken=26470c2daf5c2e2f, processorArchitecture=MSIL" />
  </configSections>
  <Depots>
    <depots>
      <add depot="MARS" />
      <add depot="JUPITER" />
      <add depot="NEPTUNE" />
    </depots>
  </Depots>
  <appSettings>
    <add key="TwinsExcludeFile" value="C:\Bin\TwinsToIgnore.txt"/>
  </appSettings>
  <system.diagnostics>
    <assert assertuienabled="false"/>
    <sources>
      <source name="EvilTwins">
        <listeners>
          <remove name="Default"/>
          <add name="AcLog" type="Microsoft.VisualBasic.Logging.FileLogTraceListener, Microsoft.VisualBasic, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL"/>
          <add name="EvilTwinsFound" type="Microsoft.VisualBasic.Logging.FileLogTraceListener, Microsoft.VisualBasic, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL"/>
        </listeners>
      </source>
    </sources>
  </system.diagnostics>
  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6.1"/>
  </startup>
</configuration>

# TwinsExcludeFile TwinsToIgnore.txt
# These evil twins will resolve when promoted to the root stream.
\.\src\com\foo.java
\.\src\bar.c
...

/* Copyright (C) 2017-2018 Verizon. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. */

// Required references: AcUtils.dll, Microsoft.VisualBasic, System, System.configuration, System.Xml.Linq
using System;
using System.Collections.Generic;
using System.Configuration;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using System.Xml.Linq;
using Microsoft.VisualBasic.Logging;
using AcUtils;

namespace EvilTwins
{
    // Compare two AccuRev elements based on their depot-relative path (case-insensitive) and element ID.
    public struct TwinEqualityComparer : IEqualityComparer<Tuple<string, int>> // [element, EID]
    {
        public bool Equals(Tuple<string, int> x, Tuple<string, int> y)
        {
            if (Object.ReferenceEquals(x, y)) // are the compared objects the same object?
                return true;
            if (Object.ReferenceEquals(x, null) || Object.ReferenceEquals(y, null)) // are either of them null?
                return false;

            bool path = String.Equals(x.Item1.ToUpperInvariant(), y.Item1.ToUpperInvariant());
            bool eid = (x.Item2 == y.Item2);
            return (path && eid);
        }

        public int GetHashCode(Tuple<string, int> elem)
        {
            int path = elem.Item1.ToUpperInvariant().GetHashCode();
            return path ^ elem.Item2; // path ^ EID
        }
    }

    class Program
    {
        #region class variables
        private static string _twinsExcludeFile; // optional TwinsExcludeFile specified in EvilTwins.exe.config
        private static List<String> _excludeList; // list of elements from TwinsExcludeFile in depot-relative format
        private static DepotsCollection _selDepots; // list of depots from EvilTwins.exe.config to query for evil twins
        private static Dictionary<AcDepot, HashSet<Tuple<string, int>>> _map; // [element, EID] for all elements in all dynamic streams in depot
        private static TwinEqualityComparer _tcompare; // help determine if two elements are evil
        private static FileLogTraceListener _tl; // logging support for evil twins that are found
        private static readonly object _locker = new object(); // token for lock keyword scope
        #endregion

        // Returns zero (0) if program ran successfully, otherwise 
        // one (1) in the event of an exception or program initialization failure.
        static int Main()
        {
            // general program startup initialization
            if (!init()) return 1;

            // if a TwinsExcludeFile was specified in EvilTwins.exe.config, 
            // validate that it exists and retrieve its content
            if (!String.IsNullOrEmpty(_twinsExcludeFile))
            {
                if (!File.Exists(_twinsExcludeFile))
                {
                    AcDebug.Log($"TwinsExcludeFile {_twinsExcludeFile} specified in EvilTwins.exe.config not found");
                    return 1;
                }
                else
                    _excludeList = getTwinsExcludeList();
            }

            // all stream types in order to include workspace streams, include hidden (removed) streams
            AcDepots depots = new AcDepots(dynamicOnly: false, includeHidden: true);
            Task<bool> dini = depots.initAsync(_selDepots);
            if (!dini.Result) return 1;

            _tcompare = new TwinEqualityComparer();
            _map = new Dictionary<AcDepot, HashSet<Tuple<string, int>>>();
            List<Task<bool>> tasks = new List<Task<bool>>(depots.Count);
            foreach (AcDepot depot in depots)
                tasks.Add(initMapAsync(depot));

            Task<bool[]> arr = Task.WhenAll(tasks); // finish running stat commands and initialization in parallel
            if (arr == null || arr.Result.Any(n => n == false)) return 1; // check log file

            Task<bool> r = reportAsync();
            return (r.Result) ? 0 : 1;
        }

        // Initialize our dictionary class variable with [element, EID] for all elements in all dynamic streams in depot. 
        // AcUtilsException caught and logged in %LOCALAPPDATA%\AcTools\Logs\EvilTwins-YYYY-MM-DD.log on stat command failure. 
        // Exception caught and logged in same for a range of exceptions.
        private static async Task<bool> initMapAsync(AcDepot depot)
        {
            bool ret = false; // assume failure
            try
            {
                IEnumerable<AcStream> filter = from n in depot.Streams
                                               where n.IsDynamic && !n.Hidden
                                               select n;
                List<Task<AcResult>> tasks = new List<Task<AcResult>>(filter.Count());
                foreach (AcStream stream in filter)
                    tasks.Add(AcCommand.runAsync($@"stat -a -s ""{stream}"" -fx")); // -a for all elements in stream

                AcResult[] arr = await Task.WhenAll(tasks); // finish running stat commands in parallel
                if (arr != null && arr.All(n => n.RetVal == 0)) // true if all initialized successfully
                {
                    HashSet<Tuple<string, int>> hset = new HashSet<Tuple<string, int>>(_tcompare);
                    foreach (AcResult r in arr)
                    {
                        // if empty the stream has an ACL that is preventing us from reading it or some other error occurred
                        if (r == null || String.IsNullOrEmpty(r.CmdResult)) continue;
                        XElement xml = XElement.Parse(r.CmdResult);
                        foreach (XElement e in xml.Elements("element"))
                        {
                            string path = (string)e.Attribute("location");
                            int eid = (int)e.Attribute("id");
                            hset.Add(new Tuple<string, int>(path, eid));
                        }
                    }

                    lock (_locker) { _map.Add(depot, hset); }
                    ret = true; // operation succeeded
                }
            }

            catch (AcUtilsException ecx)
            {
                AcDebug.Log($"AcUtilsException caught and logged in Program.initMapAsync{Environment.NewLine}{ecx.Message}");
            }

            catch (Exception ecx)
            {
                AcDebug.Log($"Exception caught and logged in Program.initMapAsync{Environment.NewLine}{ecx.Message}");
            }

            return ret;
        }

        // Report evil twins if found. Assumes the initMapAsync method has been called. Exception caught 
        // and logged in %LOCALAPPDATA%\AcTools\Logs\EvilTwins-YYYY-MM-DD.log on operation failure.
        private static async Task<bool> reportAsync()
        {
            bool ret = false; // assume failure
            try
            {
                foreach (KeyValuePair<AcDepot, HashSet<Tuple<string, int>>> pair in _map.OrderBy(n => n.Key)) // order by AcDepot
                {
                    AcDepot depot = pair.Key; // depot
                    HashSet<Tuple<string, int>> hset = pair.Value; // [element, EID] from all dynamic streams in depot
                    // from our hashset create a collection of elements mapped to their EID's
                    Lookup<string, Tuple<string, int>> col = (Lookup<string, Tuple<string, int>>)hset.ToLookup(n => n.Item1);
                    // where more than one EID exists for the element, order by element
                    foreach (var ii in col.Where(n => n.Count() > 1).OrderBy(n => n.Key))
                    {
                        string element = ii.Key;
                        if (_excludeList != null && _excludeList.Contains(element))
                            continue; // ignore if in TwinsExcludeFile

                        log(element);
                        IEnumerable<AcStream> filter = from n in depot.Streams
                                                       where n.IsDynamic && !n.Hidden
                                                       select n;
                        List<Task<XElement>> tasks = new List<Task<XElement>>(filter.Count());
                        foreach (AcStream stream in filter)
                            tasks.Add(getElementInfoAsync(stream, element));

                        XElement[] arr = await Task.WhenAll(tasks); // finish running stat commands in parallel
                        if (arr != null && arr.All(n => n != null)) // true if all ran successfully
                        {
                            foreach (Tuple<string, int> jj in ii.OrderBy(n => n.Item2)) // order twins by EID
                            {
                                int eid = jj.Item2;
                                log($"\tEID: {eid} on {DateTime.Now.ToString("ddd MMM d h:mm tt")}");
                                // C# language short-circuit: the test for id equals eid isn't evaluated if status equals "(no such elem)", 
                                // otherwise an exception would be thrown since the id attribute doesn't exist in this case
                                foreach (XElement e in arr.Where(n => (string)n.Attribute("status") != "(no such elem)" &&
                                    (int)n.Attribute("id") == eid).OrderBy(n => n.Annotation<AcStream>()))
                                {
                                    log($"\t\t{e.Annotation<AcStream>()} {(string)e.Attribute("status")}");
                                    string namedVersion = (string)e.Attribute("namedVersion"); // virtual stream name and version number
                                    string temp = (string)e.Attribute("Real");
                                    string[] real = temp.Split('\\'); // workspace stream and version numbers
                                    AcStream wkspace = depot.getStream(int.Parse(real[0])); // workspace stream
                                    ElementType elemType = e.acxType("elemType");
                                    string twin;
                                    if ((long?)e.Attribute("size") != null)
                                        twin = $"\t\t\tSize: {(long)e.Attribute("size")}, ModTime: {e.acxTime("modTime")} {{{elemType}}}" +
                                            $"{Environment.NewLine}\t\t\tReal: {wkspace}\\{real[1]}, Virtual: {namedVersion}";
                                    else // a folder or link
                                        twin = $"\t\t\tReal: {wkspace}\\{real[1]}, Virtual: {namedVersion} {{{elemType}}}";

                                    log(twin);
                                }

                                log("");
                            }
                        }
                    }
                }

                ret = true; // operation succeeded
            }

            catch (Exception ecx)
            {
                AcDebug.Log($"Exception caught and logged in Program.reportAsync{Environment.NewLine}{ecx.Message}");
            }

            return ret;
        }

        // Returns the attributes for the element in stream if the query succeeded, otherwise null. 
        // AcUtilsException caught and logged in %LOCALAPPDATA%\AcTools\Logs\EvilTwins-YYYY-MM-DD.log on stat command failure. 
        // Exception caught and logged in same for a range of exceptions.
        private static async Task<XElement> getElementInfoAsync(AcStream stream, string element)
        {
            XElement e = null;
            try
            {
                AcResult r = await AcCommand.runAsync($@"stat -fx -s ""{stream}"" ""{element}""");
                if (r != null && r.RetVal == 0)
                {
                    XElement xml = XElement.Parse(r.CmdResult);
                    e = xml.Element("element");
                    e.AddAnnotation(stream); // add stream since it's not in the XML
                }
            }

            catch (AcUtilsException ecx)
            {
                AcDebug.Log($"AcUtilsException caught and logged in Program.getElementInfoAsync{Environment.NewLine}{ecx.Message}");
            }

            catch (Exception ecx)
            {
                AcDebug.Log($"Exception caught and logged in Program.getElementInfoAsync{Environment.NewLine}{ecx.Message}");
            }

            return e;
        }

        // Returns the list of elements in TwinsExcludeFile from EvilTwins.exe.config with elements that can be ignored. 
        // Assumes caller has determined that the TwinsExcludeFile specified in EvilTwins.exe.config exists. 
        // Exception caught and logged in %LOCALAPPDATA%\AcTools\Logs\EvilTwins-YYYY-MM-DD.log on operation failure.
        private static List<String> getTwinsExcludeList()
        {
            List<String> exclude = new List<String>();
            FileStream fs = null;
            try
            {
                // constructor arguments that avoid an exception thrown when file is in use by another process
                fs = new FileStream(_twinsExcludeFile, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
                using (StreamReader sr = new StreamReader(fs))
                {
                    String line;
                    while ((line = sr.ReadLine()) != null)
                    {
                        string temp = line.Trim();
                        if (temp.Length == 0 || temp.StartsWith("#"))
                            continue; // this is an empty line or a comment
                        exclude.Add(temp);
                    }
                }
            }

            catch (Exception ecx)
            {
                AcDebug.Log($"Exception caught and logged in Program.getTwinsExcludeList{Environment.NewLine}{ecx.Message}");
            }

            finally // avoids CA2202: Do not dispose objects multiple times
            {
                if (fs != null) fs.Dispose();
            }

            return exclude;
        }

        // General program startup initialization routines.
        // Returns true if initialization was successful, false otherwise.
        private static bool init()
        {
            // initialize logging support for reporting program errors
            if (!AcDebug.initAcLogging())
            {
                Console.WriteLine("Logging support initialization failed.");
                return false;
            }

            // save an unhandled exception in log file before program terminates
            AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(AcDebug.unhandledException);

            // ensure we're logged into AccuRev
            Task<string> prncpl = AcQuery.getPrincipalAsync();
            if (String.IsNullOrEmpty(prncpl.Result))
            {
                AcDebug.Log($"Not logged into AccuRev.{Environment.NewLine}Please login and try again.");
                return false;
            }

            // initialize our class variables from EvilTwins.exe.config
            if (!initAppConfigData()) return false;
            // initialize our logging support to log evil twins found
            if (!initEvilTwinsLogging()) return false;

            return true;
        }

        // Initialize our class variables from EvilTwins.exe.config.
        // Returns true if variables were successfully initialized, false otherwise.
        // ConfigurationErrorsException caught and logged in 
        // %LOCALAPPDATA%\AcTools\Logs\EvilTwins-YYYY-MM-DD.log on initialization failure.
        private static bool initAppConfigData()
        {
            bool ret = true; // assume success
            try
            {
                _twinsExcludeFile = AcQuery.getAppConfigSetting<string>("TwinsExcludeFile").Trim();
                DepotsSection depotsConfigSection = ConfigurationManager.GetSection("Depots") as DepotsSection;
                if (depotsConfigSection == null)
                {
                    AcDebug.Log("Error in Program.initAppConfigData creating DepotsSection");
                    ret = false;
                }
                else
                    _selDepots = depotsConfigSection.Depots;
            }

            catch (ConfigurationErrorsException exc)
            {
                Process currentProcess = Process.GetCurrentProcess();
                ProcessModule pm = currentProcess.MainModule;
                AcDebug.Log($"Invalid data in {pm.ModuleName}.config{Environment.NewLine}{exc.Message}");
                ret = false;
            }

            return ret;
        }

        // Initialize our logging support for evil twins found. Output is sent to the daily log file 
        // EvilTwinsFound-YYYY-MM-DD.log created (or updated) in the same folder where EvilTwins.exe resides.
        // Returns true if logging was successfully initialized, false otherwise. Exception caught and logged in 
        // %LOCALAPPDATA%\AcTools\Logs\EvilTwins-YYYY-MM-DD.log on initialization failure.
        private static bool initEvilTwinsLogging()
        {
            bool ret = false; // assume failure
            try
            {
                TraceSource ts = new TraceSource("EvilTwins"); // this program name
                _tl = (FileLogTraceListener)ts.Listeners["EvilTwinsFound"];
                _tl.Location = LogFileLocation.ExecutableDirectory; // create the log file in the same folder as our executable
                _tl.BaseFileName = "EvilTwinsFound"; // our log file name begins with this name
                _tl.MaxFileSize = 83886080; // log file can grow to a maximum of 80 megabytes in size
                _tl.ReserveDiskSpace = 2500000000; // at least 2.3GB free disk space must exist for logging to continue
                // append -YYYY-MM-DD to the log file name
                // additional runs of this app in the same day append to this file (by default)
                _tl.LogFileCreationSchedule = LogFileCreationScheduleOption.Daily;
                _tl.AutoFlush = true; // true to make sure we capture data in the event of an exception
                ret = true;
            }

            catch (Exception exc)
            {
                AcDebug.Log($"Exception caught and logged in Program.initEvilTwinsLogging{Environment.NewLine}{exc.Message}");
            }

            return ret;
        }

        // Write text to the EvilTwinsFound-YYYY-MM-DD.log daily log file located in the same folder where EvilTwins.exe resides.
        private static void log(string text)
        {
            if (_tl != null)
                _tl.WriteLine(text);
        }
    }
}