athos
/
baangt


			
							123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368
							

<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
  <meta charset="utf-8">
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  
  <title>Beware of data generators &mdash; baangt 1.1.1 documentation</title>
  

  <script type="text/javascript" src="../_static/js/modernizr.min.js"></script>
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
        <script type="text/javascript" src="../_static/jquery.js"></script>
        <script type="text/javascript" src="../_static/underscore.js"></script>
        <script type="text/javascript" src="../_static/doctools.js"></script>
        <script type="text/javascript" src="../_static/language_data.js"></script>
    
    <script type="text/javascript" src="../_static/js/theme.js"></script>

    
  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../_static/theme_overrides.css" type="text/css" />
    <link rel="index" title="Index" href="../genindex.html" />
    <link rel="search" title="Search" href="../search.html" />
    <link rel="next" title="baangt In Industries" href="BaangtIndustries.html" />
    <link rel="prev" title="Why your production sucks and how to fix it" href="ProductionSucks.html" /> 
</head>

<body class="wy-body-for-nav">

   
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search" >
          

            <a href="../index.html" class="icon icon-home"> baangt
          

          </a>

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
              <p class="caption"><span class="caption-text">Contents:</span></p>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../Installation.html">   Installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="../OverviewUsage.html">   Overview</a></li>
<li class="toctree-l1"><a class="reference internal" href="../simpleExample.html">   First Steps</a></li>
<li class="toctree-l1"><a class="reference internal" href="../Structure.html">   Structure</a></li>
<li class="toctree-l1"><a class="reference internal" href="../ParametersConfigFile.html">   Parameters</a></li>
<li class="toctree-l1"><a class="reference internal" href="../SimpleAPI.html">   First API Test</a></li>
<li class="toctree-l1"><a class="reference internal" href="../TestTypes.html">   Types of Tests</a></li>
<li class="toctree-l1"><a class="reference internal" href="../DataFile.html">   Data file</a></li>
<li class="toctree-l1"><a class="reference internal" href="../SaveResults2Database.html">   Results in Database</a></li>
<li class="toctree-l1"><a class="reference internal" href="../HistoryAndReasons.html">   History</a></li>
<li class="toctree-l1"><a class="reference internal" href="../contributors.html">   Contributions</a></li>
<li class="toctree-l1"><a class="reference internal" href="../changelog.html">   Changelog</a></li>
<li class="toctree-l1"><a class="reference internal" href="../PlannedFeatures.html">   Planned Features</a></li>
<li class="toctree-l1"><a class="reference internal" href="../BrowserDrivers.html">   Browser Drivers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../Variables.html">   Variables</a></li>
<li class="toctree-l1"><a class="reference internal" href="../SendStatistics.html">   Results</a></li>
<li class="toctree-l1"><a class="reference internal" href="../Datagenerator.html">DataGenerator</a></li>
<li class="toctree-l1"><a class="reference internal" href="../Datagenerator.html#input-file">Input File</a></li>
<li class="toctree-l1"><a class="reference internal" href="../Datagenerator.html#data-type">Data Type</a></li>
<li class="toctree-l1"><a class="reference internal" href="../Datagenerator.html#all-data-types-format">All Data Types Format</a></li>
<li class="toctree-l1"><a class="reference internal" href="../Developer.html">   For Developers</a></li>
<li class="toctree-l1"><a class="reference internal" href="../baangt-Plugin.html">What is a baangt-plugin</a></li>
<li class="toctree-l1"><a class="reference internal" href="../baangt-Plugin.html#how-to-make-a-baangt-plugin">how to make a baangt-plugin</a></li>
<li class="toctree-l1"><a class="reference internal" href="../baangt-Plugin.html#how-the-baangt-plugin-work">how the baangt-plugin work</a></li>
<li class="toctree-l1"><a class="reference internal" href="../baangt-Plugin.html#how-to-replace-the-existing-plugin-by-your-own-one">how to replace the existing plugin by your own one</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="Articles.html">   :subheader: Articles</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="ProductionSucks.html"> Production sucks</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#"> Test data rulez</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#what-do-we-use-instead-of-random-generated-data">What do we use <strong>instead</strong> of random generated data?</a></li>
<li class="toctree-l3"><a class="reference internal" href="#that-s-all-nice-but-it-won-t-work-for-us-because">That’s all nice, but it won’t work for us because …</a></li>
<li class="toctree-l3"><a class="reference internal" href="#tl-dr">tl;dr</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="BaangtIndustries.html"> Industries 4 baangt</a></li>
<li class="toctree-l2"><a class="reference internal" href="StopTesting.html"> Stop testing!</a></li>
<li class="toctree-l2"><a class="reference internal" href="AgileWorkflowIntegration.html"> bAanGtILE</a></li>
<li class="toctree-l2"><a class="reference internal" href="BugSoup.html"> BugSoup</a></li>
<li class="toctree-l2"><a class="reference internal" href="AsynchronousAndCanonTests.html"> Canons, that are not DSLR nor music</a></li>
<li class="toctree-l2"><a class="reference internal" href="SeleniumGridV4WithBaangt.html"> SeleniumGridV4</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference external" href="http://www.baangt.org">   Web</a></li>
</ul>
<p class="caption"><span class="caption-text">Autodocs:</span></p>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../docs/baangt.base.html">Autodocs</a></li>
<li class="toctree-l1"><a class="reference internal" href="../docs/modules.html">Modules</a></li>
</ul>

            
        </div>
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../index.html">baangt</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          
<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../index.html">Docs</a> &raquo;</li>
        
          <li><a href="Articles.html">Not Exactly Documentation</a> &raquo;</li>
        
      <li>Beware of data generators</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
            
            <a href="../_sources/articles/DataDoctor.rst.txt" rel="nofollow"> View page source</a>
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
  <div class="section" id="beware-of-data-generators">
<h1>Beware of data generators<a class="headerlink" href="#beware-of-data-generators" title="Permalink to this headline">¶</a></h1>
<p>In time of big data and machine learning when we see the difficulties to find proper data for test automation we’re tempted
to turn to the silver bullet “RPA” (Robotic Process Automation). Solutions grow like mushrooms, all with the intention to
reduce effort for test data creation and test case maintenance.</p>
<p>We also use test data generators in <code class="docutils literal notranslate"><span class="pre">baangt</span></code>, very well dosed, very carefully and <strong>never</strong> to assess, whether the AUT
(Application Under Test) is suitable for production. We use test data generators based on users data recordings to harden
the application in early stages. We’ll “play” around (automatically), perform random clicks in random orders on buttons,
extend each field to it’s maximum length, fill traditional chinese characters into arabic text fields and similar tests.</p>
<p>That’s a valuable excercise. It helps to harden the application before the masses of end-users will try
to beat it down. But we use those tests not in daily or weekly regression tests or when we assess the latest build or
increment.</p>
<div class="section" id="what-do-we-use-instead-of-random-generated-data">
<h2>What do we use <strong>instead</strong> of random generated data?<a class="headerlink" href="#what-do-we-use-instead-of-random-generated-data" title="Permalink to this headline">¶</a></h2>
<p>Beautifully, precious, manually designed and recorded test data from the business department (preferably key-users or even
power end-users). We extend the dataset whenever a new issue arises in the production if in the post-mortem phase of
the issue it turns out, that regression testing could have identified the issue with proper data.</p>
<p>As this test set grows in lines, it grows in value. After a few weeks you’ve covered tests for all your major business logic
within the test set. Depending on your product cycles it might mean a hell of maintenance effort (e.g. if your pricing changes
every hour or day, you’ll either subclass the assertion methods with your own logic or you’ll be miserable updating 1000s
of comparison values in the test data), but this effort might very well lead to keeping customers (as opposed to losing them
to competition, if your software is unstable), supporting your whole organization to deliver the highest possible performance
(by having reliable software and processes in place).</p>
</div>
<div class="section" id="that-s-all-nice-but-it-won-t-work-for-us-because">
<h2>That’s all nice, but it won’t work for us because …<a class="headerlink" href="#that-s-all-nice-but-it-won-t-work-for-us-because" title="Permalink to this headline">¶</a></h2>
<p>The more you depend on software and the more complex your software and processes are, the less I believe, that it is
not possible in your case to run your regression tests with meaningful and reliable test data. Here’s a list of rather
lame excuses. If you use any of them, please think again.</p>
<table class="colwidths-given docutils align-default" id="id1">
<caption><span class="caption-text">Anti-reasons for bad test data</span><a class="headerlink" href="#id1" title="Permalink to this table">¶</a></caption>
<colgroup>
<col style="width: 10%" />
<col style="width: 40%" />
<col style="width: 50%" />
</colgroup>
<thead>
<tr class="row-odd"><th class="head"><p>Excuse</p></th>
<th class="head"><p>Rant</p></th>
<th class="head"><p>Grip</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">too</span> <span class="pre">complex</span></code></p></td>
<td><p>The software and processes are too complex. We’ve trillions of combinations. It would take months to test all that
and in the end we anyway wouldn’t know, whether the results are right or wrong.</p></td>
<td><p>System landscapes - especially grown ones - can be really complex. But at least try to find a good starting point
by combining a few, beautifully crafted test cases and validation rules with ML Data expanders. It’s better than
nothing and as your system stability improves and you need less time to fight fires in production, you’ll gain time
to invest in improving your test data. You can start an uplifting spiral!</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">no</span> <span class="pre">real</span> <span class="pre">data</span></code></p></td>
<td><p>We don’t have good quality data in our stages.</p></td>
<td><p>Often you’ll need basic or master data before you can test variations of transactional data. In a production company
you’ll need material master data of raw materials, customer master data, vendor master data and much more before
you can test your shiny new plant optimization system. But there are more benefits in investing time to create all those
prerequisit master data automated.</p>
<ul class="simple">
<li><p>You can use this high quality data also for many other processes (billing, dunning, campaigning, MRP)</p></li>
<li><p>After the next system copy, the same quality data is just one click away</p></li>
<li><p>You can produce the same quality data for test clients, migration machines or wherever else you need them</p></li>
</ul>
<p>If the mountain of work is too big to tackle it all at once, slice it down to reasonable chunks. Identify, which
data set has the highest value in terms of data quality/reliability of test case output and start with these.
A classical example are customer master data. All your test cases are with “John Doe”. First real customer record
without a ZIP-Code or strange address format for a certain country, phone number format, etc. breaks your tests.
Don’t be like that. Or at least don’t be surprised, if it happens to you. Having a few hundred different customer
master records, that you use in your test data base and recreate every now and then doesn’t take much of your time
but provides a great deal of increased reliability of your test activities.</p>
</td>
</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span class="pre">changes</span> <span class="pre">too</span> <span class="pre">fast</span></code></p></td>
<td><p>Our internal and external customers are in need for speed. We can hardly finish the features we need to provide on time,
let alone test them in E2E-Scenarios. No need for additional overhead. Our developers know, what they do!</p></td>
<td><p>Good example of “famous last words”? Regression tests don’t need to run forever. Usually one would have approx.
10-15% in pure (and slow) UI-Tests, 40% of (faster) API-Tests and rely for the rest on earlier test stages (like unit tests).
The less tests you execute, the higher the need for spotless and realistic test data! Yes, of course you can (and should)
combine various test parameters into one E2E-Testcase (e.g. a specific/problematic country code with uncommon
payment terms and a reopened order (instead of a freshly created one)). Better define a reasonable time frame for
regression tests (given the fact, that they should run automatically, maybe a nightly build gives you 2-3 hours?).
Then see, which systems take how long for each test case combination. Based on the time frame, the approximate number
of testcases and the throughput you’ll understand, if and how much you need to run in parallel. With <code class="docutils literal notranslate"><span class="pre">baangt</span></code> you’ve
the ability to run multiple TestCases in parallel and still receive exactly one result set.</p></td>
</tr>
<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span class="pre">Maintenance</span> <span class="pre">efforts</span> <span class="pre">too</span> <span class="pre">high</span></code></p></td>
<td><p>Our changes are frequent and fast. We tried to use test automation, but we ended up spending more time with the automation
then in development and in the end the reliability of the results was not worth the effort we invested.</p></td>
<td><p>Understood. It is in fact true, that UI-Tests tend to need higher maintenance efforts than API-Tests. In <code class="docutils literal notranslate"><span class="pre">baangt</span></code>
we addressed the problem of maintaining test case version with a simple and pretty versataile solution. If you need a
test case to run on different software versions, which behave differently, you can simply adjust the “Release”-field
in each Teststep (those, which are new from a certain release and those which are obsolete from a certain release, all
others remain unchanged). Instead of having to have many different versions of testcases in parallel (and potentially
the need to maintain all of them!) with <code class="docutils literal notranslate"><span class="pre">baangt</span></code> you’d have only one - unless you completely replace something
(e.g. you replace credit card payment screen <strong>completely</strong> with payment by PayPal - then you’d most probably create
a new testcase called “PayPal”).</p></td>
</tr>
<tr class="row-even"><td><p>‘Difficult to obtain data’</p></td>
<td><p>We need real data, but it’s hard to come by. We’ve validation on Phone-Numbers, IBAN, BIC and many other fields,
so we need to enter real data in test cases. But we don’t have it and we don’t want to use data from production!</p></td>
<td><p>At least you’re not using your customers data for test - that’s great! Indeed for some data it’s pretty difficult
to obtain valid test data - most of the time based on the reasoning, that if the mechanism to create valid data
was made public it would support criminal activities.</p>
<p>If you encounter such a need you’re in bad luck and your rant is granted. I assume, that your developer and API-Tests
mock this interface/data anyway, so you did the maximum possible.</p>
<p>In other cases you can use existing functionality and libraries (e.g. <code class="docutils literal notranslate"><span class="pre">baangt</span></code> support valid IBAN/BIC creation
dynamically out-of-the-box) to create valid test data and find errors long before your customers do. If you come
across the need for data, that is difficult to obtain (but legal!), contact us with a feature request on the public
issue tracker. Maybe somebody will pick it up and provide a solution in <code class="docutils literal notranslate"><span class="pre">baangt</span></code> base functionality.</p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="section" id="tl-dr">
<h2>tl;dr<a class="headerlink" href="#tl-dr" title="Permalink to this headline">¶</a></h2>
<p>Build realistic synthetic data for your regression tests and take good care of it. Also, for every bug found in production
enhance your test set!</p>
</div>
</div>


           </div>
           
          </div>
          <footer>
  
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
      
        <a href="BaangtIndustries.html" class="btn btn-neutral float-right" title="baangt In Industries" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
      
      
        <a href="ProductionSucks.html" class="btn btn-neutral float-left" title="Why your production sucks and how to fix it" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
      
    </div>
  

  <hr/>

  <div role="contentinfo">
    <p>
        &copy; Copyright 2020, Bernhard Buhl

    </p>
  </div>
  Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. 

</footer>

        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
</body>
</html>