<img height="1" width="1" src="https://www.facebook.com/tr?id=1076094119157733&amp;ev=PageView &amp;noscript=1">

Analyzing functions returning Future[T]

Posted by Saheb Motiani on Fri, Mar 3, 2017

Problem: What can you infer from looking at this function signature?

/**
 * performs a get
 */
def get(): Future[WSResponse]

I would infer that it makes an asynchronous GET request to some URL, with or without some query params, and returns a result of type Future[WSResponse].

Assumptions which I make before writing the handler:

  • It will always return a Future

  • It won’t throw exceptions, because it’s a Scala library

  • If it wants/needs to throw exceptions, it will instead return them via Future.failed(ex)

  • It will be implemented in a way that matches one of the following patterns:

def get: Future[WSResponse] = Future { . . . } 
def get: Future[WSResponse] = if (cond) Future.sucessful() else Future.failed()

Let me also tell you that it is an instance method of class WSRequest. Instances already have details about the URL they will hit (wsClient is an instance of Web Service Module).

val request: WSRequest = wSClient.url(config.url)
request.get().map { response =>
  response.status match {
    case 200 => // go ahead
    case _   => // something unexpected
  }
} recover {
    case NonFatal(ex) => // send Internal Server Error!
}

The above code should be a reasonable way to handle the response; it takes care of all the things that I knew about. We mapped over successful responses, handled different types of succesful responses, and in case the remote server is down, we recovered from Failures and responded by reporting an Internal Server Error.

I thought we were good to go, but then I got the below exception, which is a clear indication that some of my assumptions were wrong. (Maybe all of them were…)

java.lang.NullPointerException: host
    at org.asynchttpclient.util.Assertions.assertNotNull(Assertions.java:23)
    at org.asynchttpclient.uri.Uri.<init>(Uri.java:65)
    at org.asynchttpclient.uri.Uri.create(Uri.java:38)
    at org.asynchttpclient.uri.Uri.create(Uri.java:31)
    at org.asynchttpclient.RequestBuilderBase.setUrl(RequestBuilderBase.java:148)
    at play.api.libs.ws.ahc.AhcWSRequest.buildRequest(AhcWS.scala:252)
    at play.api.libs.ws.ahc.AhcWSRequest$$anon$2.execute(AhcWS.scala:166)
    at play.api.libs.ws.ahc.AhcWSRequest.execute(AhcWS.scala:168)
    at play.api.libs.ws.WSRequest$class.get(WS.scala:453)
    at play.api.libs.ws.ahc.AhcWSRequest.get(AhcWS.scala:107)

Things like this happen, but as developers we should endeavour to prevent them. Was there a way to know this could happen while I was writing the first implementation?

In case the exception wasn’t clear to you, like it wasn’t for me, the problem was with the URL, which came from a config file (and was set to something of the form "www.google.com").

Let us analyse the assumptions we made before writing the code:

  • It will always return a Future

    But what if there is some validation happening before the Future is created?

  • It can’t throw exceptions, because it’s a Scala library

    I expected no exceptions, but what if their philosophy of handling exceptions is not the same as mine? What if they believe throwing exceptions for really exceptional scenarios is okay?

    What if they are using an underlying Java library they don’t have control over?

  • If in case it wants to throw exceptions, it will instead return them via Future.failed(ex)

    But if the assumption (1) is invalid it is possible for exceptions to occur before the Future is even created.

  • It will be implemented in a way that matches one of the following patterns:

def get: Future[WSResponse] = Future { . . . } 
def get: Future[WSResponse] = if (cond) Future.sucessful() else Future.failed()

But there’s another possibility:

def get: Future[WSResponse] = {
//validation
require(url should be valid)
assertNotNull(host not null)
// Now fork a future....
Future {
   
}
// if the async operation fails then
Future.failed(new TimeoutException())
}

In this case the prior assumptions are invalidated by the use of require and assertNotNull outside of a Future.

How can we make our software resilient despite this possibility?

A number of different techniques can be used:

  • Handle it yourself for the time being by wrapping the library call in a Try block to contain the exception (solution below). You may not always be able to bring in config validation or some other more ideal solution but can instead choose to proceed with a more localised improvement.
    val request: WSRequest = wSClient.url(config.url)
    Try {
     request.get().map { response =>
       response.status match {
         case 200 => // go ahead
         case _   => // something not expected
       }
     } recover {
         case NonFatal(ex) => // send Internal Server Error!
     }
    } recover {
     case NonFatal(ex) => // Future.failed(new CustomException(s"Url Invaid! ${url}")
    }
  • Wrap the library (play web service) function call into your own safer method which can be called with more confidence, so that you know it will pass play’s hidden validation.
    /**
    * performs a get on a valid URL object
    */
    def get(uri: URI): Future[WSResponse] = {
     wsClient.url(uri.toUrl).get()
    }
  • Simply return Try[Uri] when creating URLs from strings:
    /**
    * Validates and creates a Uri from a url string
    *
    * @param url The base URL to make HTTP requests to.
    * @return either a Success(uri) or Failure(InvalidUrlException)
    */
    def url(url: String): Try[Uri] = Try(org.asynchttpclient.uri.Uri.create(url))

    Scala can be painful because it makes you deal with error cases instead of ignoring them. Hence I would have liked it more if the function signatures for the method were more type safe. That would have helped me while I was writing the calling code for the first time.

    For instance:

    def get: Either[InvalidUrlException, Future[WSResponse]] 
    def get: Try[Future[WSResponse]]

    Edit: As Future is an async version of Try, it doesn't make sense to wrap Future in a Try container. If we are improving the method signature then it is better to have it the right way, instead of nesting unnecessarily.

  • Enforce correctness via type invariants where possible, such that runtime exceptions become compile time errors (see Enforcing invariants in Scala datatypes)

    In this example the problem was not in the GET call itself, but in the execution of a GET on a WSRequest which itself wasn’t valid. Ideally it would not be possible to create the invalid WSRequest instance itself. This approach is emphasized and explained in the type invariants post).

  • Validate config during application startup, and fail fast if it’s invalid (see Config Validation using Shapeless)

In conclusion, we’ve shown:

  • Function signatures can be ambiguous. They may not be consistent across all libraries that you use.

  • The utility of Config Validation. It is better to perform validation once at the start and terminate the application if it doesn’t pass the validation criteria.

  • The importance of considering when functions might more appropriately return a Try, Either, Future or Option.

  • You can also follow up with library writers to request the change you think is for the better. They may or may not accept it, but either way, you can learn why they chose that signature in the first place.

Disclaimer: Views are my own, not my employer’s.

Thanks to Jaakko, David, Olivier and Qing for reviewing it.

Recent Posts

Posts by Topic

see all

Subscribe to Email Updates