Mar 31, 2015

Secure password storage and authentication

TLDR: If you need to store passwords, use a cryptographic function designed for password hashing like PBKDF2 or bcrypt and use a cryptographic random generator to generate a salt which is at least 64 bit long.
This article describes the common pitfalls when storing passwords and describes best practices for secure password authentication.

We accompany a young developer without the slightest clue of how to store passwords and follow him through his learning process. In each step I will describe the obvious and not-so-obvious problem with his solution. Let’s start!

Just store the passwords as plaintext

So our developer has to store the password of a user of his application. Since he knows how to work with databases, he just stores the password in plaintext.

The obvious problem with this approach is that everyone who has read access to the database can recover the passwords. As they are stored in plaintext and thus can easily be recovered, an attacker could even try to use the user’s username/password for other services, e.g. the email account.

Obfuscate the password

Our developer realizes that storing the passwords as plaintext is a bad idea. So he invents his own algorithm to obfuscate the password.

Problems with this: Cryptography is hard. There are a lot of smart guys who worked together to create secure systems, so please do not roll your own crypto. Even if the output from your self-made crypto may look secure, it probably (and that is a very high probability) is not. Also, there is a principle called security through obscurity which you should avoid.

Encrypting the password

So our developer accepts the fact that creating his own crypto-algorithm is a bad idea. He uses his search engine of choice to find examples of good crypto. And he finds the AES cipher. This cipher is widely used and generally considered as secure. So he utilizes this cipher to encrypt the passwords before storing them in the database. To encrypt the password, he needs to use a key, which is stored somewhere in the application. This key is used to decrypt the password from the database when a user wants to log on to the application.

This approach has a big problem: If the attacker has read access to the database, it is likely that he can also find the application files. And somewhere in this files lies the key to encrypt/decrypt the passwords. The attacker then just needs to read the password from the database, decrypt it with the extracted key, and voila, he gets the plaintext password.

Hashing the password

After discovering the flaw our developer reads about crypto basics and discovers a technique known as hash functions. Some well-known hash functions are MD5, SHA1 and the SHA-2xx family. A hash function is a function which takes data of an arbitrary length and maps it to a fixed-length value. Every same input data maps to the same output value. Hash functions which are used in cryptography have an interesting characteristic: they are non-invertible. That means it is very hard to find the input data for a given output value. “Hey, that’s exactly what I’m looking for”, our developer thinks. “This way I can store the password and nobody can recover the plaintext password!” Okay, you ask, but if you cannot recover the plaintext, how does the system check that the user has entered the correct password on login? Because the same input data maps to the same output value, you can take the password which the user has entered, hash it and compare it to the hash stored in the database. If they are both the same, the user has entered the correct password.

As you may have guessed, this technique also has a flaw: As every same password leads to the same hash, some guys have built big databases which do the inverse mapping, from hash to the plaintext password. As database lookups are usually really fast, the plaintext for a given hash can be looked up quite fast. Test it for yourself: Visit http://www.md5decrypt.org/ and enter some MD5 hashes. For example:

  • 5ebe2294ecd0e0f08eab7690d2a6ee69 (“secret”)
  • b295ab91a74ae7048c0ba523c03511e3 (“verysecret”)
An attacker can also use Rainbow Tables to get the plaintext to the hash.

Put in some salt

After searching the internet for some time, our developer stumbles across the recommendation to add salt to his hashes. Salt is random data which is added to the plaintext password before hashing. This defeats precomputed hashes or rainbow tables, because the same input password does not produce the same hash anymore. Every password gets its own salt, and the salt is stored in plaintext along with the password hash. When the user wants to log on to the application, the salt and the hash is read from the database. The salt is now added to the password entered from the user and the hash function is applied. Then the hash from the database is compared to the calculated hash. If both hashes match, the user is allowed to login. There are still some mistakes our developer can make:

  • Reuse the salt for every password. The salt should be unique for every password, otherwise the attacker can precalculate the hashes or build a rainbow table
  • Use a salt which is too short: If you add just, say, for example one byte of salt, the precalculated hash table grows at the factor of 256 (2 to the power of 8), but an attacker can compensate for this.
  • Use a non-random salt: If you use the username as salt for example, the salt is not random enough. Use a cryptographic random number generator (for example SecureRandom in Java) to generate the salt. If the salt isn't random, an attacker can build a rainbow table with that salt.

Performance problems the other way round

Even after using a hash function with salt, our developer recognizes, the passwords are not stored secure enough. The main problem is that hash functions are not made for password storage. Admittedly, they have the nice property of being non-invertible, but they have one flaw: they are way too fast. These hash functions have been designed to work with huge amount of data at a fast speed. With the uprising of the ability to calculate on GPUs, billions of hashes can be checked per second (!). A great tool for this is oclHashcat. An NVidia GTX 750 Ti GPU, which costs at the time of writing about 150 Euros, can create roughly 3 billion of MD5 Hashes per second. If you use a more expensive GPU or multiple GPUs, this gets even faster.

The solution, finally

After realizing that using a hash function is not the way to go when storing passwords, our developer reads about cryptographic functions which are more suitable for password storage. One of these functions is PBKDF2, the Password based keyderivation function #2. PBKDF2 applies a salt to the password and uses a hash function (or another pseudorandom function) multiple times. The number of applications of this function is called the iteration count. This technique is known as key stretching and slows down the hashing of passwords. As a result an attacker can calculate considerably fewer hashes per second. The iteration count of PBKDF2 can be adjusted to compensate the growth in computing power. If the computing power grows, just increase the iteration count. Other examples of password hashing functions are Bcrypt and Scrypt. When using such a function you need to decide on how many iterations you need. This depends on your performance requirements and your available performance. ThisStackoverflow article offers some clues.

The QAware solution

We at QAware have developed a library which handles all the unpleasant parts of secure password storage for you. The workflow looks like this:




As you can see in the code, this library has a pretty cool feature: automatic detection of broken algorithms and parameters. If you used, say, PBKDF2 with 1000 iterations, that was secure in the past, but nowadays it is not. The library will detect this and give you the possibility to update the password hash. The library also handles secure salt generation with a sufficient length and has secure defaults for iteration count and the like.

Conclusion


  • Use crypto
  • Do not roll your own crypto
  • Use a salt
  • Apply a random and long enough salt
  • Use a cryptographic function which was designed for password hashing, for example PBKDF2 or bcrypt
  • Adjust the iteration count to fit your needs

Edit 2015-06-30:

We've just made the library open source. You can download (and contribute) here: https://github.com/qaware/heimdall. Enjoy!

Mar 27, 2015

Special Purpose Iterators

This package is for the mathematically inclined only.It contains four types of methods: 

Generating iterators of Integer: There are faculty, fibonacci and hamming. While the first two are well-known, Hamming is rather particular. It is based on an exercise attributed to R. W. Hamming and reported by Dijkstra in "A discipline of programming", p. 129. The problem is this: Let p1, p2, .. pn be n integers, often but not necessarily primes. Create an iterator yielding 1 and all multiples of p1, p2, .. pn in ascending order. So, hamming(2,3,5) returns 1, 2, 3, 4, 5, 6, 8, 9, 10, 12, 15, 16, 18, ...

Generating iterators of Double: exp(), cos(), sin() generate the respective series. random(), arithmeticSeries(start, delta), geometricSeries(start, factor) behave as expected. 

Finite and Infinite Polynoms: An Iterator of Double such as (1.0, 1.0, 1.0) can be considered the finite polynom 1 + x + x^2 and hence be evaluated for any Double. The same is true for series such as exp(). The method polynom(Iterator<Double>) returns an UnaryOperator which unwinds the iterator completely or up to a given degree and stores the result in an internal list. It thus consumes the iterator but the polynom itself is reusable.
UnaryOperator<Double> p;
p = polynom(of(1.0, 2.0, 3.0));  // p(x) = 1 + 2x + 3x^2                                                                         
p.apply(0.0);                    // returns 1.0
p.apply(1.0);                    // returns 6.0
p.apply(-1.0);                   // returns 2.0
p.apply(2.0);                    // returns 17.0

p = polynom(exp());              // p(x) = exp(x)
p.apply(0.0);                    // returns 1.0
p.apply(1.0);                    // returns e
p.apply(-1.0);                   // returns 1/e
p.apply(2.0);                    // returns e^2


Operators on Polynoms Two polynoms of finite or infinite length can be added, multiplied, divided and composed, yielding in each case a new polynom. Note that the divisor's first coefficient must not be zero. compose(s, t) returns the coefficients of polynom(t) o polynom(s). It thus holds that

polynom(compose(t, s)) = compose(polynom(t), polynom(s))

Square and inverse of a polynom p are defined by
square(p) = multiply(p, p)
inverse(p) = divide(1, p)

 Everybody knows that sin^2 + cos^2 = 1. With our library, this reads as follows:
Iterator<Double> r;
r = add(square(sin()), square(cos()));
r.next();                    // first element is 1.0, all others are 0.0
reduce(limit(r, 10), (a, b) -> abs(a) + abs(b), 0.0);  // returns 0.0
An iterator can also be augmented by a scalar and multiplied with a factor:
Iterator<Double> r;
r = add(random(), 5.0);
r = multiply(random(), 5.0);

Mar 9, 2015

Handling Nulls

Functions, operators and comparators often crash on null operands, e.g. when applying map to a List containing nulls. There are at least two competing approaches to handling nulls: Optional<T> and WeakLogic.
Optional<T>, introduced with Java 8, is a wrapper: variables of type T which might be null are wrapped by an Optional<T>. This works and is clean but highly invasive because types get affected throughout the system. Equal-methods work nicely without Optional due to their different strategy: on null they return false because no non-null object ever equals null. Functions accepting null are called weak. Weak functions are less invasive than Optional because types remain unchanged.
Our class Weaklogic contains some high order functions transforming functions, operators and comparators into their weak counterpart. The idea is this: ignore null operands as long as there is at least one non-null operand, otherwise return null.
So weakEquals compares any two objects using their equals-method if possible and returning true on two nulls. Likewise weakUnaryOperator(UnaryOperator) returns a UnaryOperator identical to the argument but for its gently accepting null.
Weak comparators consider null the smallest element of the universe. weakComparator comes in two flavours: the one without argument returns a weak version of Comparable::compareTo, the other one transforms a Comparator into its weak counterpart.
So weak operators are an appealing alternative if Optional is unsuitable for whatever reason. 

Integer sum;
BinaryOperator<Integer> weakAdd = weakBinaryOperator((Integer a, Integer b) -> a + b);
sum = weakAdd.apply(null, null); // returns null
sum = weakAdd.apply(1, null); // returns 1
sum = weakAdd.apply(null, 2); // returns 2
sum = weakAdd.apply(1, 2); // returns 3
Comparator<Integer> cmp = (x, y) -> x - y;
Comparator<Integer> w = weakComparator(cmp);
w.compare(0, 1); // result < 0 w.compare(null, 1); // result < 0
w.compare(0, null); // result > 0
w.compare(null, null); // result == 0
Integer s;
Function<Integer, Integer> weak2 = weakFunction(x -> 2 * x);
s = weak2.apply(5); // returns 10
s = weak2.apply(null); // returns null

Mar 2, 2015

Android Libraries & Tools you should give a try


When we started working on our new internal project, QAwear, we had to choose some libraries to work with. QAwear is an Android app that lets you build your projects via Jenkins and Sonar and review the results - all within the app. Moreover, you can connect your smartwatch with the app. You will get notified about results and you can easily build a project: simply choose one and shake the watch ("shake to build").
We compared different libraries and these are the ones we liked most:

Retrofit
Retrofit is a type-safe REST client library for Android and Java applications, which “turns your REST Api into a Java interface”.  The request method, relative URL, parameters, headers, form encodes and multipart can be added with annotations.

For deserialization, Retrofit uses GSON by default for mapping the JSON responses to POJOs, but XML converters, Protocol Buffers or custom parsers can be configured, as well custom error handlers.

/** 
  * Common interface defintion to query Sonar and obtain data via REST. 
  */ 
public interface SonarApiServiceI { 

    /** 
      * Get a list of all projects asynchron. 

      * @param cb retrofit callback function 
      */ 
    @ GET ( "/api/resources" ) 
    void getProjectsCb ( retrofit . Callback < List < SonarProject >> cb ) ; 

    /** 
      * Get a selected sonar project with passed metrics and violations. 

      * @param project the sonar project's id 
      * @param metrics the mectrics as comma separated String 
      * @param alerts boolean, if true, violations will be returned too 
      * @return a list with one SonarProject object 
      */ 
    @GET("/api/resources") 
    List<SonarProject> getProjectMetrics (@Query("resource") int project, 
    @Query("metrics") String metrics, @Query("includealerts") Boolean alerts); 

    //... 

//generates an implementation of the SonarApiService interface. 
RestAdapter adapter = new RestAdapter.Builder()
.setEndpoint(host)
.setRequestInterceptor(new BasicAuthInterceptor(user, token)) //adding basic authentication header 
.setConverter(new GsonConverter(gson)) //setting custom JSON converter 
.setErrorHandler(new NetworkingErrorHandler(this.context)) //using custom Error handler 
.build(); 

service = adapter.create(SonarApiServiceI.class);
Other popular networking libraries are VolleyAndroid Async HTTP and Ion.


Dagger

Dagger is a dependency injection framework, which checks dependencies already at compile time and generates code to avoid reflection. This approach increases performance significantly for Android applications, because reflection is very slow in the Dalvik VM. View Demo.


Butterknife is a small library, which aims to help you write clearer, easier to understand, and less redundant code. The provided annotations turn time consuming and redundant bindings of views, onClick-Listener and other run-of-the-mill task into a pleasant one-liner. During the compilation of the application, the code for view look-ups gets generated instead of using reflection, so it doesn’t influence your app’s performance.

public class ExampleActivity extends Activity { 

    @InjectView(R.id.text_message) 
    TextView textMessage; //"injecting" TextView

    @InjectViews({R.id.user_name, R.id.password}) 
    List<EditText> credentialFields; //grouping views

    @Override public void onCreate(Bundle savedInstanceState) { 
        super.onCreate(savedInstanceState); 
        setContentView(R.layout.example_activity); 

        ButterKnife.inject(this); 

        //act on all views in the list at once
        ButterKnife.apply(credentialFields, ACTION);
    } 


    //OnClickListener 
    @OnClick(R.id.submit) void submit() { 
        //action... 


}
EventBus is a tiny and very fast Android optimized EventBus, which simplifies the communication between different components of the app.

//Defining an event
public class MyEvent () {…}

//Register subscriber
EventBus.getDefault().register(this);

//Post an event
EventBus.getDefault().post(new MyEvent());

//Receive an event – based on conventions
public void onEvent (MyEvent event) {…};

//Unregistering
EventBus.getDefault().unregister(this);
The SDK Manager Plugin is a Gradle plugin that downloads and updates missing or out-of-date SDKs and support libraries automatically before building your app. It ensures, that all developer and even your CI system have all necessary SDKs and support libraries to be able to build your application.
You only have to add the dependency for the SDK Manager and apply the plugin before the regular 'android' plugin to your build.gradle file:
buildscript { 

    repositories { 
        mavenCentral() 
    } 

    dependencies 
        classpath 'com.android.tools.build:gradle:0.12.+' 
        classpath 'com.jakewharton.sdkmanager:gradle-plugin:0.12.+' 
   } 



apply plugin: 'android-sdk-manager' 
apply plugin: 'com.android.application'

Robolectric is a powerful unit testing framework, with the significant benefit of reducing the overhead of deploying apps on a real device or an emulator before every test depending on Android Api components. By replacing all Android classes with so-called shadow objects, behaving similar to the objects of the Android SDK, you can even run your tests inside the regular JVM. You don’t depend on a Dalvik VM and its long test run startup time.
Furthermore, it is possible to provide custom implementations for specific SDK methods for simulating error conditions or hardware features like real-world sensors and behavior, for example. View Demo.


Fabric.io

Fabric was announced 2014 at twitter’s first developer conference. By the time of this post it’s still in a closed beta phase, but already one of the best tools for crash reporting and beta distribution (advertising platform not tried).
Fabric gets shipped out of the box, you only have to install the Plugin for your IDE and select the components you want to use in your app. The Plugin will show you, which dependencies have to be added and the positions in your code where changes have to be made and will perform them for you.
The crash reports are very detailed. You can see all information of the affected device, that could help you to retrace the bug’s cause as well as the whole stack trace of the uncaught exceptions over all threads.
Furthermore it provides extensive user statistics and beta distribution, similar to Google Analytics and Google play beta tests.