Monday, September 24, 2018

Java SE 9... What's New? [Code Samples - 02/02]

As I had promised, albeit very late, I am back with the Code Samples of my earlier article Java SE 9... What's New? You may refer the earlier article to understand the newly introduced features at a High Level. This article provides the Concluding Part of Code Samples for each of the Features. You may refer to the First Part of this Article at Java SE 9... What's New? [Code Samples - 01/02]

You may Download the Code Samples for the following New Features Here. (Import as Eclipse Project, Set Compiler/Environment as Java 9. Run the Main Class 'Java9Application' to see the Output/Outcome of the Code Samples.)


B. Core Library Changes in Java 9 (Continued)

09. Stack-Walking API
Prior to Java 9, The way to access Stack Trace was very limited and provided the entire dump or stack information at once. This was inefficient and did not allow any direct way of filtering data. With Java 9, a Lazy StackWalker API has been introducedThis will allow to fetch data based on filtering conditions and is more efficient. 
 package com.techilashots.java9.features;  
 import java.lang.StackWalker.Option;  
 import java.util.List;  
 import java.util.stream.Collectors;  

 /**  
  * Stack Walker API is a new feature of Java 9, aimed at Improving Performance of the predecessor Stack Track Element,  
  * as also for providing a way to filter the Stack Elements, in case of Exception or to Understand Application Behavior.   
  * Although, there have been multiple changes, I am covering only Stack Frame Attributes and also the walk() method   
  * for Walking the Stack Frame.   
  */  
 public class StackWalkingService { 
 
      private int databaseService() {  

           int x = 3;  

           // Usage 01: Walking All Stack Frames  
           System.out.println("Java 9 Stack Walker API - Showing All Stack Frames");  
           StackWalker stackWalker = StackWalker.getInstance();  
           stackWalker.forEach(System.out::println);  
           System.out.println("");  

           // Usage 02 : Filtering or Walking Stack Frames  
           System.out.println("Java 9 Stack Walker API - Walking / Filtering Stack Frames");  
           List<StackWalker.StackFrame> stackFrames;  
           stackFrames = stackWalker.walk(frames -> frames.limit(2).collect(Collectors.toList()));  
           stackFrames.forEach(System.out::println);  
           System.out.println("");  

           // Usage 03 : Show All Attributes of a Stack Frame  
           System.out.println("Java 9 Stack Walker API - Show All Attributes in Stack Frame");  
           StackWalker newWalker = StackWalker.getInstance(Option.RETAIN_CLASS_REFERENCE);  
           stackFrames = newWalker.walk(frames -> frames.limit(1).collect(Collectors.toList()));  
           stackFrames.forEach(sfRecord->   
           {  
                System.out.printf("[Bytecode Index] %d%n", sfRecord.getByteCodeIndex());  
                System.out.printf("[Class Name] %s%n", sfRecord.getClassName());  
                System.out.printf("[Declaring Class] %s%n", sfRecord.getDeclaringClass());  
                System.out.printf("[File Name] %s%n", sfRecord.getFileName());  
                System.out.printf("[Method Name] %s%n", sfRecord.getFileName());  
                System.out.printf("[Is Native] %b%n", sfRecord.isNativeMethod());  
                System.out.printf("[Line Number] %d%n", sfRecord.getLineNumber());  
           });  
           return x;  
      }  

      private float persistenceService() {  
           float x = databaseService();  
           return x;  
      }  

      private double businessService() {  
           double x = persistenceService();  
           return x;  
      }  

      private double presentationService() {  
           long x = (long) businessService();  
           return x;  
      }  

      public void uiDisplay() {  
           System.out.println("Java 9 Stack Walker API for Debugging and Application Behavior");  
           double x = presentationService();  
           System.out.println("\n[Method to Display On User Interface]");  
           System.out.println("Old MacDonald had a Farm. In that Farm, He had " + x + " Cows!");  
      }  

      /**  
       * @param args  
       */  
      public static void main(String[] args) {  
           StackWalkingService stackWalkingService = new StackWalkingService();  
           stackWalkingService.uiDisplay();  
      }  
 }  


10. Compact Strings
Although this has no external ramification to a developer in terms of syntax or semantics change - It may impact the way we design for memory and performance. The current UTF-16 representation uses 2 Bytes for Storage. Most of the string contain characters that are only Latin-1 in nature. The Latin-1 characters require only 1 Byte for Storage. With Java 9, String storage has been modified to start with an Additional Encoding Flag. This flag indicates whether it contains ISO-8859-1/Latin-1 characters or the UTF-16 characters. As per the official word, It has lead to an Improved Usage of Memory and Efficient GC, but with Some Loss in Performance at Peak Loads.

The Compact Strings are always enabled in Java 9, but it can be disabled by passing in the VM Argument +XX:-CompactStrings

It has to be noted that in Java 9,  the implementation of java.lang.String decides at runtime, whether the storage size is to be 2 Bytes or 1 Byte, as per the actual String size (UTF-16 or Latin-1 Character).


11. Spin-Wait Hints
For multi-threading applications, this brings in some performance improvements under Busy-Waiting or Spin-Waiting conditions. Usually, Busy-Waiting is done for synchronization of some state of the object between two or more invokers - Waiting for a condition to occur before processing starts or continues. Thread.onSpinWait() has been introduced as a static method in the Thread class and can be optionally called in Busy-Waiting loops - This will allow the JVM to issue processor instructions on some system architectures to improve Reaction Time in such Spin-Wait Loops and also Reduce the Power Consumed by the Core Thread or Hardware thread. This benefits the Overall Power Consumption of a Program, and possibly Allowing other Cores or Hardware Threads to execute at Faster Speeds within the Same Power Consumption Envelope.
 package com.techilashots.java9.features;  

 import java.util.List;  
 import java.util.Vector;  

 /**  
  * For a Single Line Demonstration, I wrote the Non-Pure Threaded form of Producer-Consumer. Run the Code Multiple Times  
  * On a Machine, where you can Understand how the Temperature Changes and Extra Cooling Fan Kicks Off. Even though I did  
  * not do it myself, Take it up as an Experiment, by removing the Thread.onSpinWait() [Compare Before/After and also try  
  * with Data Set of 1 Million to 10 Million]  
  *   
  * Thread.onSpinWait() will Optimize Latency and Reduce/Optimize Power Consumption  
  */  
 public class SpinWaitHints {  

      public static void main(String[] args) {  
           List<String> itemQueue = new Vector<String>();  
           Producer producer = new Producer(itemQueue);  
           Consumer consumer = new Consumer(itemQueue);  
           producer.start();  
           consumer.start();  
      }  
 }  
 class Producer extends Thread {  

      List<String> itemQueue;  
      Producer(List<String> itemQueue) {  
           this.itemQueue = itemQueue;  
      }  
      public void run() {  
           try {  
                produce();  
           } catch (InterruptedException e) {  
                e.printStackTrace();  
           }  
      }  
      public void produce() throws InterruptedException {  
           while (true) {  
                while (itemQueue.size() < 100000) // produce 1 lac items  
                {  
                     String item = "Sumith Puri " + (itemQueue.size());  
                     itemQueue.add(item);  
                     System.out.println("Item Produced: " + item);  
                }  
                while (itemQueue.size() > 0) {  
                     // spin waiting - x86 architectures will now optimize  
                     Thread.onSpinWait();  
                }  
           }  
      }  
 }  

 class Consumer extends Thread {  
      List<String> itemQueue;  
      public Consumer(List<String> itemQueue) {  
           this.itemQueue = itemQueue;  
      }  
      public void consume() throws InterruptedException {  
           while (true) {  
                while (itemQueue.size() < 100000) {  
                     // spin waiting - x86 architectures will now optimize  
                     Thread.onSpinWait();  
                }  
                int x = itemQueue.size();  
                while (x >= 1) {  
                     x = x - 1;  
                     if (x >= 0) {  
                          String item = itemQueue.remove(x);  
                          System.out.println("Item Consumed: " + item);  
                     }  
                }  
                if (itemQueue.size() > 0)  
                     itemQueue.remove(0);  
           }  
      }  

      public void run() {  
           try {  
                consume();  
           } catch (InterruptedException e) {  
                e.printStackTrace();  
           }  
      }  
 }  


12. New Version-String Scheme
From Java 9, $MAJOR.$MINOR.$SECURITY.$PATCH  is the naming scheme for releases in Java. These details are also contained in the Runtime.Version Class.
 package com.techilashots.java9.features;  
 /**  
  * $MAJOR.$MINOR.$SECURITY+$BUILD is the Naming Scheme for Version String in Java.   
  */  
 public class JavaVersionStringChanges {  
      public void printVersionInformation() {  
           Runtime.Version versionInfo = Runtime.version();  
           System.out.println("Version String Changes in Java 9");  
           System.out.println("Major Version: " + versionInfo.major());  
           System.out.println("Minor Version: " + versionInfo.minor());  
           System.out.println("Security Version: " + versionInfo.security());  
           System.out.println("Build Version: " + versionInfo.build());  
           System.out.println("\nIn Java 9, Version Naming is Major.Minor.Security.Build");  
           System.out.println("Currently Running in Java Version: " + versionInfo.toString());  
      }  
      public static void main(String[] args) {  
           new JavaVersionStringChanges().printVersionInformation();  
      }  
 }  


13. Enhanced Method Handles
A method handle is a typed, directly executable reference to an underlying method, constructor, field, or similar low-level operation, with optional transformations of arguments or return values. These transformations are quite general, and include such patterns as conversioninsertiondeletion, and substitution. In Java 9, Method Handles have been Enhanced to include static methods for creating different kind of Method Handles.

 package com.techilashots.java9.features;  
 import java.lang.invoke.MethodHandle;  
 import java.lang.invoke.MethodHandles;  

 /**  
  * MethodHandles were introduced first in Java 7. You have to think them as an alternative for Java Reflection API, but  
  * with an advantage of better performance as they are specified at creation time. Enhanced Method Handles has primarily  
  * added new static methods to better/widen the functionality provided by Method Handles.  
  *   
  * Note that Method Handles are Enhanced in Java 9, to introduce very many changes and methods. I will be covering the  
  * ones that are the most important, only to introduce this topic.   
  *   
  * arrayLength, arrayConstructor, zero, empty,   
  * loop, countedloop, iteratedloop, dowhileloop, whileloop, try/finally  
  */  
 public class EnhancedMethodHandles {  

      public void enhancedMethodHandleDemo() {  

           try { 
 
                // arrayLenth  
                MethodHandle methodHandleLength = MethodHandles.arrayLength(int[].class);  
                int[] ageArray = new int[] { 21, 28, 36 };  
                int arrayLength;  
                arrayLength = (int) methodHandleLength.invoke(ageArray);  
                System.out.println("Length of Array using Method Handle is " + arrayLength);  
                
                // arrayConstructor  
                MethodHandle methodHandleConstructor = MethodHandles.arrayConstructor(int[].class);  
                int[] newAgeArray = (int[]) methodHandleConstructor.invoke(3);  
                System.out.println("Array Constructed using Method Handle of Size " + newAgeArray.length);  
                
                // zero  
                int x = (int) MethodHandles.zero(int.class).invoke();  
                System.out.println("Default Value of Primitive Integer using Method Handles is " + x);  
                String y = (String) MethodHandles.zero(String.class).invoke();  
                System.out.println("Default Value of String using Method Handles is " + y);  
                
                System.out.println();  
                System.out.println("Reader/Developer - Left as an Exercise for You :-)");  
                System.out.println("Refer Loop, CountedLoop, DoWhileLoop, WhileLoop, IteratedLoop, TryFinally");  
           } catch (Throwable e) {  
                
                System.out.println("Was Hungry as Ever - Gulped Everything I Got!");  
           }  
           // refer to, https://goo.gl/JCyo7N (official javadoc)  
           // also use, https://goo.gl/i8wNJ8 (individual blog)  
      }  

      public static void main(String[] args) {  
           new EnhancedMethodHandles().enhancedMethodHandleDemo();  
      }  
 }  



14. Variable Handles
Java's Concurrent Package (java.util.concurrent.atomic) provide all Atomic Types for performing atomic operations. Apart from this,  Unsafe Operations (sun.misc.unsafe) such as creating objects without calling the constructor used in Java Low-Level Programming require to be hidden from the outside world as per JEP 260: Encapsulate Most Internal APIs This has led to creation of a new abstract class type named VarHandle - This will allow a developer to assign different types to the same reference (dynamically typed reference). It can also take care of performing atomic operations on the held variable, including compare and swap (set or exchange) operations. It also provides memory fencing operations, to order the in-memory representation of the object, by providing finer grain control. 

Firstly, You need to understand Memory Ordering Effects, as VarHandle are entirely based on the Understanding of Plain, Volatile, Opaque, Acquire/Release Memory Ordering Modes. You may refer to them at : https://www.baeldung.com/java-variable-handles (Under Memory Ordering Effects). You have to try and visualize this correctly, and then proceed to the Code Samples.
 package com.techilashots.java9.features;  

 import java.lang.invoke.MethodHandles;  
 import java.lang.invoke.VarHandle;  

 /**  
  * VarHandle allows developers to assign different types to the same reference (dynamically typed reference).It can also   
  * take care of performing atomic operations on the held variable, including compare and swap (set/exchange) operations.   
  * It also provides memory fencing operations, to order the in-memory representation of the object, by providing finer   
  * grain control.   
  *   
  * I am providing an Example of the Read Operations using VarHandle. Please refer to the link provided below for further  
  * info on VarHandle on Public Variable, VarHandle for Private Variables, VarHandle for Array Types   
  *   
  * https://www.baeldung.com/java-variable-handles  
  */  
 class VarHandleStore {  

      public int varIntHandle01 = 5;  
      public int varIntHandle02 = 9;  
      public byte varByteHandle03 = 21;  
 }  

 public class VariableHandles {  

      public void useVariableHandle() {  

           System.out.println("Java 9 Introduces.... Variable Handles!");  

           try { 

                VarHandleStore varHandleStore = new VarHandleStore();  
                VarHandle varHandle = MethodHandles.lookup().in(VarHandleStore.class).findVarHandle(VarHandleStore.class,  
                          "varIntHandle01", int.class);  
                // value using get() in varhandle  
                int plainValue = (int) varHandle.get(varHandleStore);  
                System.out.println("Value using get() in VarHandle: " + plainValue);  

                // value is written to using set() - plain access  
                // you can also use set(), setOpaque(), setVolatile(), setRelease()  
                varHandle.set(varHandleStore, 21);  
                plainValue = (int) varHandle.get(varHandleStore);  
                System.out.println("Set Value using set(), then get() in VarHandle: " + plainValue);  
                
                // value is written to using getandadd()  
                int oldValue = (int) varHandle.getAndAdd(varHandleStore, 51);  
                plainValue = (int) varHandle.get(varHandleStore);  
                System.out.println("Using getAndAdd() in VarHandle, Old Value: " + oldValue + ", New Value: " + plainValue);  
                varHandle = MethodHandles.lookup().in(VarHandleStore.class).findVarHandle(VarHandleStore.class,  
                          "varIntHandle02", int.class);  
                
                // please do try out the compareandset() - atomic updates  
                // have left this due to time constraints  
                // value is written to using getandbitwiseor()  
                varHandle = MethodHandles.lookup().in(VarHandleStore.class).findVarHandle(VarHandleStore.class,  
                          "varByteHandle03", byte.class);  
                byte before = (byte) varHandle.getAndBitwiseOr(varHandleStore, (byte) 127);  
                byte after = (byte) varHandle.get(varHandleStore);  
                
                System.out.println("Get Byte Value, Then Or, using getAndBitwiseOr()");  
                System.out.println("Old Byte Value: " + before + "; New Byte Value: " + after);  
           } catch (NoSuchFieldException | IllegalAccessException e) {  
 
                e.printStackTrace();  
           }  
      }  

      public static void main(String[] args) {  
           new VariableHandles().useVariableHandle();  
      }  
 }  


15. Filter Incoming Serialization Data
This feature is related to the addition of filters at the serialization incoming streams to improve security and robustness. The core mechanism is a filter interface implemented by serialization clients and set on an ObjectInputStream. The filter interface methods are called during the de-serialization process to validate the classes being de-serialized, the sizes of arrays being created, and metrics describing stream length, stream depth, and number of references as the stream is being decoded. The filter returns a status to accept, reject, or leave the status undecided. Some Helpful Information On this Feature can be Found at: https://goo.gl/bRezWt You have to remember the words Security, Vulnerabilities and Robustness as the main reasons behind creation of this Feature in Java. This will thwart possible Security Attacks like Denial Of Service.

The core mechanism is a Filter Interface implemented by serialization clients and set on an ObjectInputStream. The filter interface methods are called during the Deserialization Process to Validate the classes being Deserialized, The Sizes of Arrays being Created, The Stream Length, Graph Depth and Number of References as the Stream is being Decoded. A filter determines whether the arguments are ALLOWED or REJECTED and should return the appropriate status. If the filter cannot determine the status it should return UNDECIDED. Filters are designed for the specific use case and expected types. A filter designed for a particular use may be passed a class that is outside of the scope of the filter. If, for example, the purpose of the filter is to Black-List Classes then it can reject a candidate class that matches and report UNDECIDED for others.
 package com.techilashots.java9.features;  

 import java.io.ObjectInputFilter;  

 /**  
  * Demonstrates Java 9 Serialization/De-Serialization Filters for Incoming Data. Do Refer https://goo.gl/bRezWt for more  
  * on Filters, with more Details and Examples.  
  */  
 class ItemCatalogFilter implements ObjectInputFilter {  

      private long maxStreamBytes = 400; // Maximum allowed bytes in the stream.  
      private long maxDepth = 2; // Maximum depth of the graph allowed.  
      private long maxReferences = 5; // Maximum number of references in a graph.  

      @Override  
      public Status checkInput(FilterInfo filterInfo) {  

           if (filterInfo.references() < 0 || filterInfo.depth() < 0 || filterInfo.streamBytes() < 0  
                     || filterInfo.references() > maxReferences || filterInfo.depth() > maxDepth  
                     || filterInfo.streamBytes() > maxStreamBytes) {  
                // reject this, as it seems malicious, incorrect or tampered with  
                return Status.REJECTED;  
           }  

           Class<?> clazz = filterInfo.serialClass();  
           if (clazz != null) {  
                if (CatalogCustomer.class == filterInfo.serialClass()) {  
                     // we are expecting a customer of our product catalog  
                     System.out.println("Incoming Serialization Data Verified for Structure and Vulnerabilities");  
                     return Status.ALLOWED;  
                } else {  
                     // seems like some tampered, unexpected or malicious data here  
                     return Status.REJECTED;  
                }  
           }  

           // the status as undecided, when we cannot infer - or + for sure  
           // left for the developer to decide as per business/security process  
           return Status.UNDECIDED;  
      }  
 }  


You may Download the Code Samples for the following New Features Here. (Import as Eclipse Project, Set the Compiler/Environment as Java 9. Make Sure that you Have Java 9 / JDK 9 Installed on your System. Run the Main Class 'Java9Application' to see the Output/Outcome of the Code Samples.)

Check out the Eclipse Console Output provided Below from Running the 'Java9Application' (Refer Attached Code Samples). Please Refer to the Individual Classes, provided for Each of the Features to Understand the Features Better. Go Ahead - Add, Modify, Delete to Experiment with all of the New Java 9 Features. The Output for the Features Marked [01-08] is  Shown in the Part 1 of the Article at 
Java SE 9... What's New? [Code Samples - 01/02]


 ================================  

 09. Stack Walking API  
 ---------------------  
 Java 9 Stack Walker API for Debugging and Application Behavior  
 Java 9 Stack Walker API - Showing All Stack Frames  
 com.techilashots.java9.features.StackWalkingService.databaseService(StackWalkingService.java:23)  
 com.techilashots.java9.features.StackWalkingService.persistenceService(StackWalkingService.java:57)  
 com.techilashots.java9.features.StackWalkingService.businessService(StackWalkingService.java:63)  
 com.techilashots.java9.features.StackWalkingService.presentationService(StackWalkingService.java:69)  
 com.techilashots.java9.features.StackWalkingService.uiDisplay(StackWalkingService.java:77)  
 com.techilashots.java9.main.Java9Application.stackWalkingAPI(Java9Application.java:321)  
 com.techilashots.java9.main.Java9Application.java9CoreLibraryChanges(Java9Application.java:106)  
 com.techilashots.java9.main.Java9Application.main(Java9Application.java:48)  
 Java 9 Stack Walker API - Walking / Filtering Stack Frames  
 com.techilashots.java9.features.StackWalkingService.databaseService(StackWalkingService.java:30)  
 com.techilashots.java9.features.StackWalkingService.persistenceService(StackWalkingService.java:57)  
 Java 9 Stack Walker API - Show All Attributes in Stack Frame  
 [Bytecode Index] 112  
 [Class Name] com.techilashots.java9.features.StackWalkingService  
 [Declaring Class] class com.techilashots.java9.features.StackWalkingService  
 [File Name] StackWalkingService.java  
 [Method Name] StackWalkingService.java  
 [Is Native] false  
 [Line Number] 38  
 [Method to Display On User Interface]  
 Old MacDonald had a Farm. In that Farm, He had 3.0 Cows!  

 ================================  

 11. Spin Wait Hints  
 -------------------  
 Spin Wait Hints Makes Power Consumption Efficient  
 [Uncomment Method Call in Code, To Run (Non-Terminating) Demo]  

 ================================  

 12. Java Version Naming  
 ----------------------  
 Java 9 changes the way Java Version String Format  
 Version String Changes in Java 9  
 Major Version: 9  
 Minor Version: 0  
 Security Version: 4  
 Build Version: Optional[11]  
 In Java 9, Version Naming is Major.Minor.Security.Build  
 Currently Running in Java Version: 9.0.4+11  

 ================================  

 13. Enhanced Method Handles  
 ---------------------------  
 Java 9 has Enhanced Method Handles for Multitude of Operations  
 Length of Array using Method Handle is 3  
 Array Constructed using Method Handle of Size 3  
 Default Value of Primitive Integer using Method Handles is 0  
 Default Value of String using Method Handles is null  
 Reader/Developer - Left as an Exercise for You :-)  
 Refer Loop, CountedLoop, DoWhileLoop, WhileLoop, IteratedLoop, TryFinally  

 ================================  

 14. Variable Handles  
 --------------------  
 Java 9 Introduces Variable Handles, Different Types for Same Reference  
 Java 9 Introduces.... Variable Handles!  
 Value using get() in VarHandle: 5  
 Set Value using set(), then get() in VarHandle: 21  
 Using getAndAdd() in VarHandle, Old Value: 21, New Value: 72  
 Get Byte Value, Then Or, using getAndBitwiseOr()  
 Old Byte Value: 21; New Byte Value: 127  

 ================================  

 15. Filter Incoming Serialization Data  
 -------------------------------------  
 Java 9 Allows Filtering of Serialization Data, through Object Input Filters  
 Incoming Serialization Data Verified for Structure and Vulnerabilities  
 Customer Details String Follows (Filtered then De-Serialized)  
 skp:skp:4240123488889001:1.243223256E7  

 ================================  


_______________________________________________________________________________

16. More Concurrency Updates
[Will be Covered In a Separate Article, As there are Many Changes to Cover and Merits an Article on Its Own]
_________________________________________________________________________________

[Note that I will be covering 'Java 9 Modularity - Project Jigsaw' in another Article. The following Topics/Changes with Code Samples will be Covered in that Article]

JEP 261: Module System
JEP 200: The Modular JDK 
JEP 220: Modular Run-Time Images JEP 260: Encapsulate Most Internal APIs   _________________________________________________________________________________


Happy Coding with Java 9!
 
 

Tuesday, September 18, 2018

Java SE 9... What's New? [Code Samples - 01/02]

As I had promised, albeit very late, I am back with the Code Samples of my earlier article Java SE 9... What's New? You may refer the earlier article to understand the newly introduced features at a High Level. This article provides the Code Samples for each of the Features.

You may Download the Code Samples for the following New Features Here. (Import as Eclipse Project, Set Compiler/Environment as Java 9. Run the Main Class 'Java9Application' to see the Output/Outcome of the Code Samples.)

I plan to start by re-visiting a Java 8 Feature 'Default Interface Methods'


00. Default Interface Methods
Whenever there is existing or legacy code which has Interfaces that require addition of new methods - It causes breakage of existing classes that inherit/implement from this Interface unless the implementation for each of these added methods is provided in the classes. This is not very maintainable code. Even though a good practice as per SOLID and other OO paradigms to provide an interface without any implementation - we need to handle and solve the problem as mentioned above. This is where Default Interface Methods come in.

01: import java.util.List;  
02:    
03:  public interface LegacyPublicInterface {  
04:        
05:        
06:      /**  
07:       * Additional Default Method that Can be Invoked on Each Type of Invoice that Implements the LegacyPublicInterface.   
08:       * It can also be over-ridden by the Extending Invoice Types. This is an Example Usage and Benefit of the Java/JDK 8  
09:       * Default Interface feature.  
10:       * @param items  
11:       */  
12:      default void checkStatusOnEachItem (List<String>... items) {  
13:        
14:          for(String item : items) {  
15:              if(item.startsWith("OOS_")) {  
16:                  items.remove(item);  
17:              }  
18:          }  
19:          return;  
20:      }  
21:    
22:  }  
23:    


From the Example Above, the LegacyPublicInterface in an Existing Application is already extended by Multiple Invoice Types (For Example in an Inventory System). But as per changing business, Re-Engineering effort requires that each of the Invoices have a method to invalidate or remove an Item marked with "OOS". Given such a problem, Prior to Java 8, we would have to introduce a new method declaration in the interface and then require that each of the implementing classes implement their own logic for handling this. With Default Interfaces, the task becomes very simple (The code is now more maintainable and extensible and requires very less effort to change). With the introduction of thus feature of Default Methods, the following are the possibilities:

1. Use the Default Method(s) without Breaking Existing Functionality (Best Use)
2. Implementing Class can choose to override these Default Methods
3. Abstract Classes can be provided over the Interfaces to override Implementation

So, Let's get further into each of these changes in Java 9, one-by-one. 


A. Java Language Changes in JDK 9

[01] Private Interface Methods
Interfaces in Java 9 are allowed to have Private Methods. This was done to allow code sharing between non-abstract methods in the Interface. All rules related to ordinary Private modifier apply to these methods. The point to note is that a method cannot be both private and abstract. It definitely needs to have a method body.

 package com.techilashots.java9.features;  
 
/**  
  * @author sumith.puri  
  *   
  * prior to java 8, the default scope/visibilty of every method in an interface was same  
  * the attributes were all by default, [public static final] and the methods were [public]  
  * with java 8, [default] interface methods have been introduced but so has been a new scope/visibility  
  * with java 9, interface can have private visibility methods that can be used internally   
  */  
 public interface Java9PrivateMethodInterface {  

      public void preJava9BusinessMethodOne();  

      public void preJava9BusinessMethodTwo();  

      public default void newJava9BusinessMethodThree() {  
           System.out.println("Interface Methods can call Private Interface Methods!");  
           System.out.println("");  
           java9BusinessMethodThreeHelper();  
      }  

      private void java9BusinessMethodThreeHelper() {  
           System.out.println("Default Methods... Now Private Methods!");  
           System.out.println("Once Upon a Time, I was a Java Developer!");  
      }  
 }  


You can observe the usage of the above private method (internally by the new default method) through the following code.

      //new features of java 9  
      //private interface methods  
      private void privateInterfaceMethod() {  

           System.out.println("01. Private Interface Methods");  
           System.out.println("-----------------------------");  
           System.out.println("[Private Interface Method Invoked by Default Interface Method]");  
           Java9PrivateMethodInterface java9PrivateMethodInterface = new Java9PrivateMethodInterface() {  
                @Override  
                public void preJava9BusinessMethodTwo() {  
                     // TODO Auto-generated method stub  
                }  
                @Override  
                public void preJava9BusinessMethodOne() {  
                     // TODO Auto-generated method stub  
                }  
           };  

           java9PrivateMethodInterface.newJava9BusinessMethodThree();  
           System.out.println("");  
           System.out.println("================================");  
           System.out.println("");  
      }  


[02] Underscore as a Variable Name is not Legal Anymore
Using only the _ (underscore character) as a variable name is not legal anymore.This is because it is marked as a reserved keyword from Java 1.8 (But causing compilation failure only in Java 1.9). This may cause a some issues when compiling legacy source code, especially which had a necessity to denote some specific resource or entity using the _ (underscore).  It may have to be rewritten and may have many related ramifications.
 /**  
  * @author sumith.puri  
  *  
  */  
 public class Java9VariableNaming {  
      // pre-java 9, this was a valid variable name  
      // from java 8, _ was marked as a reserved keyword (compiler warning) 
      // from java 9, the following line of code will cause compilation failure  
      // private String _;  
      private String _socket = null;  
      /**  
       * @param args  
       */  
      public void notForDemoMethod() {  
           _socket = new String("Network Socket");  
           System.out.println("_ is no Longer a Valid Variable Name!, [Variable Value: " + _socket + "]");            
      }  
 }  


[03] Allow Effectively final variables to be used as Resources in Try with Resources
Upto Java 8, Every variable that had to be used within Try with Resources statements requires to be declared within the try statement. Only then, can it be used within the try block. This is a limitation for th developer. Hence, In Java 9, this restriction has been removed and any final variable or effectively final (local) variable can be used inside the try block. All other rules as applicable to Try with Resources continue. Effectively Final means the variable that is not changed once after it has been initialized.

 package com.techilashots.java9.features; 

 import java.io.BufferedReader;  
 import java.io.File;  
 import java.io.FileReader;  
 import java.io.IOException;  
 
 /**  
  * @author sumith.puri  
  */  
 public class EffectivelyFinalTryResources {  
 
     private static File file = new File("try_resources.j9");   

      //with java 9, you need to use either explicitly final or effectively final variables in try/resources  
      public void methodWithAnomaly() throws IOException {  

           file.createNewFile();  
           BufferedReader bufferedReader = new BufferedReader(new FileReader(file));  
       
           //prior to java 9, the usage would look like this  
           //try (final BufferedReader bufferedReader = new BufferedReader(new FileReader(file))) {  
           //this code will not compile before java 9  
           try (bufferedReader) {            
                System.out.println("Can Use Final or Effectively Final in Try with Resources!");  
           } finally {  
                System.out.println("Java 9 Gives More Flexibility to Developers.");  
           }            
      }  
 }  


[04] @SafeVarargs is Allowed on Private Instance Methods
Using non-reifiable variable argument parameters in a method can cause multiple warnings when trying to compile code, such as:

Note: LegacyPublicInterface.java uses unchecked or unsafe operations.  
Note: Recompile with -Xlint:unchecked for details.

Hence @SafeVarargs was introduced to suppress such warnings on unchecked or unsafe operations. By using this, the developer is signalling to the compiler that he has made sure that there will be no Heap Pollution (such as unsafe forEach operations) caused.

Prior to Java 9, @SafeVarargs is allowed on non-overridable methods such as in static methods, final instance methods and constructors. Note that the annotation will throw an error if it is used in fixed arity methods.  In Java 9, @SafeVarargs can be used on private instance methods.
 package com.techilashots.java9.features;  

 /**  
  * @author sumith.puri  
  */  
 public class SafeVarargsOnPrivateMethod {  

      public void demoSafeVarargsInJava9() {  
           safeVarargsInJava9(24.00f, 03, 19, 82);  
      }  

      @SafeVarargs  
      private void safeVarargsInJava9 (Float a, Integer...b) {  
           System.out.println("Invoked a Private Instance Method with " + a + ", " + b[0] + ", " + b[1] + ", " + b[2]);  
           System.out.println("With Java 9, @SafeVarargs is Allowed on Private Instance Methods!");  
      }  
 }  


[05] Allow Diamond with Anonymous Classes If Inferred Type's Argument Type is Denotable
Upto Java 8, using Generics and Diamond Operators with Anonymous Classes. It was mainly because the compiler could not infer whether it can represent the type in the Argument passed to the Diamond Operator.  JSR 334 has the following to say about using diamond with anonymous classes:
"Using diamond with anonymous inner classes is not supported since doing so in general would require extensions to the class file signature attribute to represent non-denotable types, a de facto JVM Change." 

Additional information is located in the Project Coin mailing list's Diamond Operator and Anonymous Classes topic:
"Internally, a Java compiler operates over a richer set of types than those that can be written down explicitly in a Java program. The compiler-internal types which cannot be written in a Java program are called non-denotable types. Non-denotable types can occur as the result of the inference used by diamond. Therefore, using diamond with anonymous inner classes is not supported since doing so in general would require extensions to the class file signature attribute to represent non-denotable types, a de facto JVM change. It is feasible that future platform versions could allow use of diamond when creating an anonymous inner class as long as the inferred type was denotable."
With Java 9, the Java Compiler has changed its Inference Algorithm in a way that Diamond Operator (Generics) can now work simultaneously with Anonymous Classes, as long as the Argument Type of the Inferred Type is Denotable. The important point to note is that things that fall under Denotable are Primitive Types, Raw Types and Non-Generic Types. Non-Denotable means ones that cannot be written in a Java Program, like usage of Extends, Super along with Wildcard Types in Generics. These are usually inferred by the compiler. So, as long as the compiler identifies that the Argument Type of Inferred Type is Denotable - You can use the Diamond Operator in Conjunction with Anonymous Inner Classes.


 package com.techilashots.java9.features;  
 /**  
  * @author sumith.puri  
  */  
 public class DiamondOperatorOnAnonymous {  
      public void diamondForInferredDenotable() {  
           //prior to java 9, anonymous inner classes were not allowed to use diamond operator  
           Java5ToJava8Coder<? extends Number> java9SyntaxGuru = new Java5ToJava8Coder<>() {  
                     @Override  
                     public void java9SyntaxMagic() {  
                          System.out.println("Introduced in Java 5, Generics was Intriguing and Complex!");  
                          System.out.println("With Java 9, Diamond Operator On Anonymous Classes is Allowed...");  
                          System.out.println("As Long as Java 9 Infers the Type (Inferred) to be Denotable.");  
                     }  
           };  
           java9SyntaxGuru.java9SyntaxMagic();  
      }  
 }  
 abstract class Java5ToJava8Coder<T extends Number> {  
       public abstract void java9SyntaxMagic();  
 }


B. Core Library Changes in Java 9

[06] JEP 102: Process API Updates
With Java 9, one can retrieve the PID of the process through a native call. This is achievable through the ProcessHandle. Also, we can retrieve information about the currently running Java Process (JVM) and Info (Inner Class of ProcessHandle) Class/Object that contains details on the process. We can also enlist or return a snapshot of all currently running processes in the system.
 package com.techilashots.java9.features;  
 import java.lang.ProcessHandle.Info;  

 /**  
  * @author sumith.puri  
  */  
 public class ProcessAPIChanges {  
      
      public void detailedAPIInfo(ProcessHandle processHandle) {  
           
           Info processInfo = processHandle.info();  
           System.out.println("[Java 9 Developers... Check this Out!]");  
           System.out.println("[Detailed Process Info is Provided Below]");  
           System.out.println("[Executable Name] " + processInfo.command().get());  
           System.out.println("[User Name] " + processInfo.user().get());  
           System.out.println("[Start Time] " + processInfo.startInstant().get().toString());  
           System.out.println("+++++");  
      }
  
      public static void main(String[] args) {  
           
           System.out.println("06. Process API Changes (Core Library) ");  
           System.out.println("--------------------------------------");  
           System.out.println("Check Out the Detailed Process Information in Java 9");  
           
           ProcessAPIChanges processAPIChanges = new ProcessAPIChanges();  
           ProcessHandle processHandle = ProcessHandle.current();  
           
           System.out.println("With Java 9, Process Id is Available");  
           System.out.println("[Current Process Id] " + processHandle.pid());  
           System.out.println("-------------------------------------");  
           
           processAPIChanges.detailedAPIInfo(processHandle);  
           
           System.out.println("-------------------------------------");  
           System.out.println("With Java, You can View all Processes..");  
           System.out.println("That are Visible to the Current Process!");  
           
           ProcessHandle.allProcesses()  
            .filter(ph -> ph.info().command().isPresent())  
            .limit(4)  
            .forEach((process) -> processAPIChanges.detailedAPIInfo(process));  
           
           System.out.println("");  
           System.out.println("================================");  
           System.out.println("");  
      }  
 }  


[07] JEP 277: Enhanced Deprecation
With an eye on maintainable and more informative code, the Developer defined Deprecation now allows us to mark deprecation with additional elements of information like forRemoval and since. The forRemoval allows allows to mark that this item may be removed in the future versions of Java and since provides information about when it was first introduced.

 package com.techilashots.java9.features;  
 
 /**  
  * @author sumith.puri  
  */  
 public class EnhancedDeprecation {  
      
      @Deprecated(forRemoval=true)  
      public void methodMarkedForRemoval() {  
           System.out.println("Java 9 Allows Enhanced Method Deprecation");  
           System.out.println("Invoked Method is Deprecated and Marked [For Removal]");  
           this.methodDeprecatedSince();  
      }  
      
      @Deprecated(since="12.2")  
      public void methodDeprecatedSince() {  
           System.out.println("Invoked Method is Deprecated and Marked [Since]");  
      }  
 }  
 
 

[08] JEP 269: Convenience Factory Methods for Collections
This addition makes it convenient for the Developer to create Immutable Collections out of existing interfaces, be it Set, Map or List. A static factory method of() added to each of the interfaces - Set, Map and List. It is important you understand the following (even though consistent with previous versions of Java):
  • They are Structurally Immutable
  • They Disallow Null elements or null keys. 
  • They are Serializable if all elements are serializable.
  • They Reject Duplicate Elements/Keys at creation time. 
  • The Iteration Order of set elements is Unspecified and is Subject to Change.
  • They are Value-Based. Factories are free to create new instances or reuse existing ones. Therefore, Identity-Sensitive Operations on these instances (Reference Equality (==), Identity Hash Code, and Synchronization) are Unreliable and should be Avoided.They are serialized as specified on the Serialized Form page.
 package com.techilashots.java9.features;  
 import java.util.Set;  
 /**  
  * @author sumith.puri  
  */  
 public class ConvenienceCollections {  
      public void checkItOut() {  
           System.out.println("Java 9 Introduced a Static [of()] Factory Method");  
           System.out.println("This allows Creation of Immutable Collections");  
           Set<String> immutableCountrySet = Set.of("America", "Russia", "China", "India");  
           try {  
                immutableCountrySet.add("England");  
           } catch (Exception e) {  
                System.out.println("Caught Exception, Adding Entry to Immutable Collection!");   
           }  
      }   
 }  


You may Download the Code Samples for the following New Features Here. (Import as Eclipse Project, Set the Compiler/Environment as Java 9. Make Sure that you Have Java 9 / JDK 9 Installed on your System. Run the Main Class 'Java9Application' to see the Output/Outcome of the Code Samples.)

Check out the Eclipse Console Output provided Below from Running the 'Java9Application' (Refer Attached Code Samples). Please Refer to the Individual Classes, provided for Each of the Features to Understand the Features Better. Go Ahead - Add, Modify, Delete to Experiment with all of the New Java 9 Features.


 Sample of New Features of Java 9  
 ================================  
   
 00. Default Interface Methods  
 -----------------------------  
 [Already Integrated Java Service - Default Implementation]  
 Core Business Method One  
 Core Business Method Two  
 Java 8 Changed the Definition of Interface Forever!  
 Thou shall not clear the Basic Core Java Interview  
   
 [Newly Integrated Java Service - Override Default Implementation]  
 Core Business Method One  
 Core Business Method Two  
 Attention - All Java Developers - Default Interfaces  
 The World is now a Newer Place from Java 8!  
   
 ================================  
   
 01. Private Interface Methods  
 -----------------------------  
 [Private Interface Method Invoked by Default Interface Method]  
 Interface Methods can call Private Interface Methods!  
   
 Default Methods... Now Private Methods!  
 Once Upon a Time, I was a Java Developer!  
   
 ================================  
   
 02. Underscore Variable Naming  
 -----------------------------  
 [Nothing to Demonstrate - Refer Code for New Rule]  
 _ is no Longer a Valid Variable Name!, [Variable Value: Network Socket]  
   
 ================================  
   
 03. Effectively Final Try with Resources  
 ----------------------------------------  
 [Nothing to Demonstrate - Refer Code for New Rule]  
 Can Use Final or Effectively Final in Try with Resources!  
 Java 9 Gives More Flexibility to Developers.  
   
 ================================  
   
 04. Safe Varargs on Private Method  
 ----------------------------------  
 [Nothing to Demonstrate - Refer Code for New Rule]  
 Invoked a Private Instance Method with 24.0, 3, 19, 82  
 With Java 9, @SafeVarargs is Allowed on Private Instance Methods!  
   
 ================================  
   
 05. Diamond on Anonymous Classes   
 --------------------------------  
 [Nothing to Demonstrate - Refer Code for New Rule]  
 Introduced in Java 5, Generics was Intriguing and Complex!  
 With Java 9, Diamond Operator On Anonymous Classes is Allowed...  
 As Long as Java 9 Infers the Type (Inferred) to be Denotable.  
   
 ================================  
   
 06. Process API Changes (Core Library)   
 --------------------------------------  
 Check Out the Detailed Process Information in Java 9  
 With Java 9, Process Id is Available  
 [Current Process Id] 1380  
 -------------------------------------  
 [Java 9 Developers... Check this Out!]  
 [Detailed Process Info is Provided Below]  
 [Executable Name] C:\Users\U074SXP\Documents\My Installed\jdk-9.0.4\bin\javaw.exe  
 [User Name] CWT\U074SXP  
 [Start Time] 2018-09-18T00:21:57.354Z  
 +++++  
 -------------------------------------  
 With Java, You can View all Processes..  
 That are Visible to the Current Process!  
 [Java 9 Developers... Check this Out!]  
 [Detailed Process Info is Provided Below]  
 [Executable Name] C:\Windows\System32\taskhost.exe  
 [User Name] CWT\U074SXP  
 [Start Time] 2018-09-17T22:39:11.949Z  
 +++++  
 [Java 9 Developers... Check this Out!]  
 [Detailed Process Info is Provided Below]  
 [Executable Name] C:\Program Files\Microsoft Application Virtualization\Client\AppVStreamingUX.exe  
 [User Name] CWT\U074SXP  
 [Start Time] 2018-09-17T22:39:11.996Z  
 +++++  
 [Java 9 Developers... Check this Out!]  
 [Detailed Process Info is Provided Below]  
 [Executable Name] C:\Program Files (x86)\Symantec\Symantec Endpoint Protection\12.1.7004.6500.105\Bin\ccSvcHst.exe  
 [User Name] CWT\U074SXP  
 [Start Time] 2018-09-17T22:39:12.074Z  
 +++++  
 [Java 9 Developers... Check this Out!]  
 [Detailed Process Info is Provided Below]  
 [Executable Name] C:\Program Files\Synaptics\SynTP\SynTPEnh.exe  
 [User Name] CWT\U074SXP  
 [Start Time] 2018-09-17T22:39:12.214Z  
 +++++  
   
 ================================  
   
 07. Enhanced Deprecation  
 ------------------------  
 [Nothing to Demonstrate - Refer Code for New Rule]  
 Java 9 Allows Enhanced Method Deprecation  
 Invoked Method is Deprecated and Marked [For Removal]  
 Invoked Method is Deprecated and Marked [Since]  
   
 ================================  
   
 08. Convenience Collection Method  
 ---------------------------------  
 [Nothing to Demonstrate - Refer Code for New Rule]  
 Java 9 Introduced a Static [of()] Factory Method  
 This allows Creation of Immutable Collections  
 Caught Exception, Adding Entry to Immutable Collection!  
   
 ================================  


Happy Coding with Java 9!
 

Tuesday, September 4, 2018

Agile and Scrum Basics

I have about 15+ years of Software Development Experience (2018) and have been working on Agile Projects since about 2006. My First Formal Introduction into Agile was through a Training by Thoughtworkers (Thoughtworks is a Leading Agile Company). This was while I was a Senior Software Engineer at Huawei, Bangalore, India. I have worked on Agile/TDD/Pair Programming (Various Variants) in multiple companies including Huawei, Symantec, Yahoo, Finastra*, Oracle*, OpenText*. Recently and Once Again, I attended a Formal Classroom Training (Company Internal) on Agile. I jotted down the most important points and now am presenting it in this Blog. I hope it helps and becomes a Ready Reckoner for Understanding/Learning the Agile Basics (Needs, Motivations, Practice and Evolution Story).

* [Original Product Firms were Acquired by these Current Companies]

We will start with Dilbert!


[ IMAGE COPYRIGHT, IF ANY, WITH RESPECTIVE OWNER - OBTAINED FROM GOOGLE SEARCH ]
 


 
The Waterfall Accident 
Formal Software Development, specifically for who started into the field in 1990s or even 1980s was essentially a very Heavy Plan Driven Approach. The name given to it was Waterfall Software Development Process (As a Result of Being a Set of Successive Steps akin to a Waterfall). Even Academia captures it in textbooks with the name as the Waterfall Process. The proponent of this approach was an American Computer Scientist by the name Winston Royce. In his 1970s paper, he first formally defined the Stage Wise Waterfall Process (Though he is not the one who named it 'Waterfall'). The sequence wise phases that were identified by him are as follows:
  1. Requirements (System)
  2. Requirements (Software)
  3. Analysis
  4. Design
  5. Development (Coding)
  6. Testing
  7. Operations

[CREDITS BEGIN: WIKIPEDIA/OTHERS]
Royce called them "Implementation Steps to Develop a Large Computer Program for Delivery to a Customer".  


[ IMAGE COPYRIGHT, IF ANY, WITH RESPECTIVE OWNER - OBTAINED FROM GOOGLE SEARCH ]


Royce actually foresaw a major shortcoming in this methodology, which he described as:
The testing phase which occurs at the end of the development cycle is the first event for which timing,storage, input/output transfers, etc., are experienced as distinguished from analyzed. These phenomena are not precisely analyzable. They are not the solutions to the standard partial differential equations of mathematical physics for instance. Yet if these phenomena fail to satisfy the various external constraints,then invariably a major redesign is required. A simple octal patch or redo of some isolated code will not fix these kinds of difficulties. The required design changes are likely to be so disruptive that the software requirements upon which the design is based and which provides the rationale for everything are violated.

According to Royce; In the Process Model "The Design Iterations are never Confined to the Successive Step", and for that model without iteration is "Risky and Invites Failure". As alternative Royce proposed a more Incremental Development, where every next step links back to the step before.
[CREDITS END: WIKIPEDIA/OTHERS]

 Sometime Later and Ignoring the Shortcoming stated by Royce, the American Department of Defense accepted this "Waterfall Model" as the Standard for Defense System Software Development.  It promoted the One-Pass Document-Driven Waterfall process. It contained sentences like “The Contractor shall establish the Top-Level Design of each CSCI by Allocating Requirements from the SRS and, if applicable, IRS to the TLCSCs of each CSCI.” This was the beginning of the real widespread usage, application and practice of the "Waterfall Model". This model was drafted under the document DoD STD 2167. The Principal Author of this document later regretted promoting such a rigid process. He also agreed that he was not familiar with a Iterative, Evolutionary Requirements Based or Incremental Development Process. Even though, In 1987 DoD started warning against the "Waterfall Model" - It was in such widespread usage that Organizations all over the World refused to update their Process in their Software Development Documents and/or in Real Practice. The above events related to the Waterfall Process or the Original/First Formal Software Development Process are often termed as the The Waterfall Accident.

Shortcomings of Waterfall (Visualization
As Everywhere Else, I am representing the Real-World Situation in Usual Software Project Development (Aggravated by The Waterfall Model). 


[ IMAGE COPYRIGHT, IF ANY, WITH RESPECTIVE OWNER - OBTAINED FROM GOOGLE SEARCH ]

There is this one more diagram, that also emphasizes the reason as to why the above situation is created. It is primarily due to the rigidity in the process and also the barriers to move back and forth between various phases. The diagram is good in proving that without iterative back and forth movements between the core phases, neither changing requirements can be incorporated reliably nor can any newly identified constraints.


[ IMAGE COPYRIGHT, IF ANY, WITH RESPECTIVE OWNER - OBTAINED FROM GOOGLE SEARCH ]


 

Agile Manifesto (What is Agile?)
In 2001, group of 17 Software Thinkers and Practitioners met in Utah to arrive at a set of four common values that led to the creation of the Agile Manifesto (http://agilemanifesto.org/). I am quoting from this site directly, 

We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:

Individuals and Interactions over Processes and Tools
Working Software over Comprehensive Documentation
Customer Collaboration over Contract Negotiation
Responding to Change over Following a Plan

That is, while there is value in the items on
the right, we value the items on the left more.

This was the beginning of the Agile Movement / Revolution, that now is the Software Development Process of choice across the world.



Twelve Principles of Agile
The above mentioned group of 17 people had also chalked out the Twelve Driving Principles for Teams that Adopt Agile. You may find the actual 12 principles at the following link: http://agilemanifesto.org/principles.html

  1. Customer Satisfaction by Early and Continuous Delivery of Valuable Software
  2. Welcome Changing Requirements, even in Late Development
  3. Working Software is delivered Frequently (Weeks rather than Months)
  4. Close, Daily Cooperation between Business People and Developers
  5. Projects are built around Motivated Individuals, who should be Trusted
  6. Face-to- Face Conversation is the Best Form of Communication (Co-Location)
  7. Working Software is the Primary Measure of Progress
  8. Sustainable Development, able to maintain a Constant Pace
  9. Continuous Attention to Technical Excellence and Good Design
  10. Simplicity— the Art of Maximizing the Amount of Work Not Done—is Essential
  11. Best Architectures, Requirements, and Designs emerge from Self-Organizing Teams
  12. Regularly, the Team Reflects on how to become More Effective & Adjusts Accordingly


Visualizing Agile
Since we were discussing about Waterfall, you should first understand how the Agile Process can/should be Visualized. This will allow you to understand the Flexibility, Adaptive, Iterative and Incremental Nature of Agile Methodology.

[ IMAGE COPYRIGHT, IF ANY, WITH RESPECTIVE OWNER - OBTAINED FROM GOOGLE SEARCH ]



Also, from the perspective of the Stakeholder(s) Developing and Delivering the Software, It can/should be visualized as the following. This is to understand that the Chunks of Work are better manageable than getting bogged down by the Big Block of Work. It also should allow to Increase Quality, Manage Timelines, Continuous Delivery and  Achieve Greater Customer (For Whom the Work Is Done) Satisfaction.




[ IMAGE COPYRIGHT, IF ANY, WITH RESPECTIVE OWNER - OBTAINED FROM GOOGLE SEARCH ]



Original Agile Signatories (Who Started Agile?)
I am also mentioning the names of these original 17 Software Developers who were the original signatories of the Agile Manifesto. They should continue to find special mention in years to come, as they were the ones instrumental in Formalization of such a  game-changing Software Development Process. Together, they published the Agile Manifesto for Software Development

Kent Beck
Mike Beedle
Arie van Bennekum
Alistair Cockburn
Ward Cunningham
Martin Fowler
James Grenning
Jim Highsmith
Andrew Hunt
Ron Jeffries
Jon Kern
Brian Marick
Robert C. Martin
Steve Mellor
Ken Schwaber
Jeff Sutherland
Dave Thomas


Types (Practice, Frameworks) of Agile (How to Agile?)
Since Agile only provides Values and Principles to guide Development Teams, there are many Methods or Methodologies through which Agile is Implemented. Each of them may require an article of their own. Also, I am familiar with only about 5-6 of these Methodologies to really be able to write about them comprehensively. I am enlisting first the Software Development Frameworks through which Agile can be Implemented. I am providing only the Topmost Implementation Methods, as per my selection. There may be many more ways to Implement Agile, with some or the other Innovative Value Add being done to the Agile Process every now and then. I have also provided the Wikipedia Links so that you may directly read about them.


Apart from the above mentioned Agile Development Methods, there are multiple keywords and practices that are associated with Agile Development. I am only mentioning the ones that I am most familiar with, along with their direct Wikipedia Links.



Scrum Basics
The term Scrum was first mentioned in 1986 Harvard Business Review Article by  Hirotaka Takeuchi and Ikujiro Nonaka. They compared High Performing Cross-Functional Teams to Scrum Formation in Rugby. Scrum is a way to Implement Agile and teams working are called as Scrum Teams. The Five Values that should Drive Scrum Teams are Below:

Focus: Because we Focus on only a Few Things at a time, we work Well Together and produce Excellent Work. We deliver Valuable Items sooner. 
Courage: Because we work as a Team, we feel supported and have more resources at our Disposal. This gives us the courage to Undertake Greater Challenges.
Openness: As we Work Together, We Express How We're Doing, What's in Our Way? and Our Concerns, so they can be addressed.
Commitment: Because we have Great Control over our Own Destiny, we are more Committed to Success.
Respect: As we Work Together, Sharing Successes and Failures, We come to Respect each other and to help each other become Worthy of Respect. 


Three Pillars of Scrum that are Fundamental to Scrum Include the Following:

Transparency: Advocates that the Significant Parts of the Process to be Visible to All.
Inspection: Scrum Artefacts Constantly Inspected as also Progress towards Milestones.
Adaptation: Deviation of any Process Aspects Outside Acceptable Limits must be Adjusted.




Scrum Roles  
The real world Implementation of Agile through Scrum has Three Important Roles. 

Development Team/Member
Takes on Tasks and Delivers Chunks of Work, In Frequent Increments

Scrum Master
Protects the Scrum Process and Prevents any Distractions.

Product Owner
Determines what Needs to be Done and Sets Priorities to Deliver the Highest Value





Companies Adopting Agile
I myself have worked for Top Software Companies of the World including Yahoo (Altaba), Symantec, Huawei, Siebel (Oracle), GXS (OpenText) and Misys (Finastra). Also, I have worked for some well known IT Services/Consulting Companies like Infosys, Headstrong (Genpact) and also relatively smaller Product Brands like Persistent and Aptean. Since I started my career in 2003, I saw the move towards Agile Adoption (Variants, Loose Variants) in Various Companies throughout my Experience - It was really exciting for me as a Software Developer since the greatest crib that I ever had was Excessive Software Documentation and the Lack of Energy and Excitement. The idea of Regular Delivery of Working Software (Demos) was inherent and natural to me as a Software Developer. It is the one that brings the Greatest Joy (Apart from Everything, as Discussed in Earlier Sections) in Software Development. I am enlisting few of the Other Companies that have Adopted Agile (or are Agile Proponents), across Business Lines.

  • Google
  • Microsoft
  • ThoughtWorks
  • CA Technologies
  •  Barclays
  • Ericsson

From my Own Experience, I can comfortably say that almost all companies in the world now use Agile/Scrum (and/or Variants) as one the Software Development Methodology of Choice. Also, Most Software Development Companies (Read Product Software) were always almost Agile, but now may be using the Formal Agile Principles as their Driving Factor to achieve Greater Efficiency. Many Companies use Variants of Agile or Mix of Agile with Other Processes for their Teams. Agile is also not restricted to Software Development - There are Industries and Functions that now use Formal Agile Methods for their Deliverables and Daily Tasks.


 


Popular Tools for Agile/Scrum
You will come across these Tools and Technologies that are used to Implement or Drive Agile/Scrum Methods and Practices in Organizations Worldwide. There may be many more Tools, Frameworks, Technologies - But I am only enumerating either the ones that I have come across or the Ones that are Popular. They may not be directly Agile Tools, but ones that either Accelerate Agile, Used for Agile Project Management, Agile Planning, Agile Task Management and Continuous Integration/Delivery.
  • Rally Agile Platform
  • Atlassian JIRA
  • (Microsoft) GitHub
  • Apache Jenkins
  • Zoho Sprints
  • Visual Paradigm
  • Thoughtworks Slack
  • Thoughtworks Mingle
  • Microsoft Team Foundation Server



Agile Pitfalls (Reasons for Failure)
Since we are now familiar with the very basics of Agile and Scrum, It is time to move on and find out the Statistics in Real-Time. These statistics reveal that even though Agile is no silver bullet, but it does provide a very high project success rate. Usually, Companies mix Agile with Other Existing or Newer Development Processes to try and increase success rate, as per the Context. 

 [CREDITS: IMAGE IS COPYRIGHTED TO VITALITY CHICAGO]


This article will try and cover the reasons for the failure of Agile Projects - From the above statistics we can easily observe that the Project Success Rate of 42% in Agile is almost 1.6 times that of Waterfall. At the same time the ratio of Failure or Challenged Projects is 1 : 1.27 (Agile : Waterfall).

 
[CREDITS: IMAGE IS COPYRIGHTED TO VITALITY CHICAGO]
 

The Best Candidates for Waterfall are Large to Very Large Projects which have very Static Requirements and with little or no External Factors or Systems impacting them. In such cases, Waterfall has greater chances of yielding Higher Quality Output in possibly  than Agile. 

Next, I am enumerating the reasons for Failures of Agile Projects, themselves. Since, It is a Highly Dynamic and Adaptive Process, I am always a strong believer that Lack of Highly Skilled Workers as one Top Reason for Failure of Agile Projects. With Agile, the Time in Utilizing Current Skills and Experience is the Most Important Differentiating Factor in Success of a Project. There is a Constant Focus and Pressure of Constant Delivery that the Holistic Skill Gain from Agile has to be through Individualistic Discipline and Practice. The Outcome of Agile in Short Term is again, creation of Lowly Skilled or Semi Skilled Workers. The Task of Gaining In-Depth Knowledge or creating Absolute Experts is no longer Automatically Supported. This may Impact Quality of the Deliverables and Software itself. But then, through Individual Practice, Character and Discipline, the Individual Engineer or Developer may be different at a Skill Levels from others, even at the Same Experience Level. So, I am only enumerating the reasons here for failure, as an In-Depth discussion is beyond the Scope of this Article. You may read these reasons, more in detail at the following link:
https://www.agilealliance.org/8-reasons-why-agile-projects-fail/

  1. Insufficient Training / Low Skill Level
  2. Unwillingness of Team to Follow Agile
  3. Broader Organizational or Communications Problems
  4. Lack of Support for Cultural Transition
  5. Pressure to Follow Traditional Waterfall Process
  6. Lack of Management Support (Transitioning to Agile)
  7. Company Philosophy at Odds (Versus Agile)  
  8. Lack of Experience in Agile (Transitioning to Agile)


Agile/Scrum in Real-World (2018)
So, The thoughts presented in this write-up are an Introduction to Agile and Scrum. In real-time, What is it that is really followed/practiced in Agile Projects? I am mentioning this from my Own Experience of Working in Agile Projects (and Variants). If you have never worked in an Agile (or Variants) Project, you may find the following in an Agile Project. It is not that all of this never existed in Waterfall, but mentioning this in a (best-effort) sequence that you may come across in an Agile Development Project.

  1. Project Kick-Off
  2. Project Milestones
  3. Sprint Planning
  4. Backlog
  5. Acceptance Criteria
  6. Story (T-Shirt) Sizing 
  7. Fibonacci Scale 
  8. Sprint Tasks
  9. Capacity
  10. Daily Scrum
  11. User Story
  12. Technical Story
  13. Technical Spike
  14. Research Spike
  15. Architectural Spike
  16. Refactoring Spike
  17. Burn- Down
  18. Burn- Up 
  19. Velocity 
  20. Sprint Demo
  21. Sprint Review
  22. Sprint Retrospective
  23. Project Retrospective
  24. Go-Live

I myself discovered these two separate manifesto's that are directly aligned to Agile Movement - One of them provides guidelines for Software Developers and other for Project Management. I am mentioning few salient points and the links where you can read more on these. In the real-world, as per me, you may not come across these as being used directly - but possibly being practiced without knowledge of such a Formal Manifesto.




Software Craftmanship Manifesto (Software Developers/Craftsmen)
I am quoting from the site directly, http://manifesto.softwarecraftsmanship.org/ - You may visit the link and also sign the Software Craftmanship Manifesto.

As Aspiring Software Craftsmen we are raising the bar of Professional Software Development by practicing it and helping others learn the craft. Through this work we have come to value: 

Not Only Working Software, But also Well-Crafted Software
Not Only Responding to Change, but also Steadily Adding Value
Not Only Individuals and Interactions, but also a Community of Professionals
Not Only Customer Collaboration, but also Productive Partnerships  

 That is, in pursuit of the items on the left we have found the items on the right to be indispensable.


After reading this manifesto for the first time, I strongly feel that it has taken the Agile Manifesto rightfully ahead and should be a mandatory read for every Software Developer. The folks who are being introduced to Agile as Software Developers, should thoroughly note the points and make it a part of their Daily Activities and Tasks.



Declaration of Interdependence (Project Management)
Continuing on the Agile Manifesto to align Management Principles that are required to achieve an Agile Mindset in Product and Project Management. The Six Principles that are essential to Modern Project Management. You may read more on this Declaration at the following link: https://en.wikipedia.org/wiki/PM_Declaration_of_Interdependence
Increase Return on Investment by making Continuous Flow of Value our Focus. 
Deliver Reliable Results by engaging Customers in Frequent Interactions and Shared Ownership. 
Expect Uncertainty and manage for it through Iterations, Anticipation and Adaptation. 
Unleash Creativity and Innovation by recognizing that Individuals are the Ultimate Source of Value...
Boost Performance through group accountability for Results and Shared Responsibility for Team Effectiveness. 
Improve Effectiveness and Reliability through situationally Specific Strategies, Processes and Practice


The above Declaration of Interdependence outlines the Leadership Approaches used to manage the Inter Dependency of People, Processes and Value to enhance the Performance of Work. The Six Key Areas are:
  1. Focus on Value
  2. Engage Customers
  3. Expect Uncertainty
  4. Unleash Creativity
  5. Group Accountability
  6. Improve Effectiveness

 
Happy Agile Development to All!