Optional Shouldn’t Be Optional

The Optional Horror Show

I’ve been reading-up on Java 8’s Optional<T> class and it’s proper usage. It might not surprise you that opinions on its correct usage vary greatly and passionately.

  • Only for return values never arguments
  • Use it everywhere!
  • Never as a field value
  • Only for return values…but only sometimes
  • Not for all getters…but some
  • Let’s make it serializable!

Even the experts don’t have consensus. So why was optional introduced? As I understand it, its primary purpose was to make the stream API more fluent and to return a reasonable result from terminal stream operations. Terminal operations need to return a result even when a collection is empty. For example:

x = s.findFirst().or(valueIfEmpty)

Nicolai Parlog wrote an excellent article discussing the background of Optional. I suggest everyone read it. It gives a great summary of the back and forth that went on around the introduction of Optional<T>.

I will use Optional<T> but ultimately I think it was a misstep. My primary issue with Optional is that it is a wrapper class that must be instantiated. Because of this I have to decide when to use it and when not to use it. This will lead to many different styles of using Optional and to an overall proliferation of use. For example:

  • Do we use Set<Optional<T>> and Map<K, Optional<V>>? Some people will and some people wont.
  • Do we have Java Bean getters return Optional<T>? If the answer is “sometimes” then how is that not a problem?
  • Because Optional<T> is a wrapper class it cannot simply implement Serializable – if it did it would have indeterminate behavior when it wraps a non-serializable class.
  • Most of the existing Java SDK classes do not and will not use Optional<T>. So I will have to accommodate to styles anyway.

With Optional as a wrapper class so many frameworks have been impacted and thus need to change. For example:

  • JSON, XML, JPA, etc. libraries need to consider Optional in order to properly read and write values from getters that return an Optional<T>.
  • Bean Validation needs to unwrap Optional<T> and validate its value…but wait, what if we want @NotNull bean constraint on the Optional itself and not it’s wrapped value?
  • Existing Java SDK classes send an unclear message on optional. They do not use Optional<T> so now we have to take care ANYWAY when using return values.

The decision to introduce Optional as a wrapper class has a far-reaching impact and it’s usage will proliferate and, worse yet, proliferate in an inconsistent way. At this point I do not think the Genie can be put back into the lamp.

Do Alternatives Exist?

If you think about it, Optional is really just behavior it is not state. For example, Optional allows us to write fluent code like this:

String value =
   Optional.ofNullable(map.get("foo"))
      .orElse("bar");

Optional behavior does require any properties outside of the object it is operating on. Now, I am not a language designer. I assume everyone that worked on defining Lambda is much smarter than I am. Nonetheless, I cannot wrap my head around the decision to introduce the Optional<T> wrapper. Wouldn’t life have been easier (even if done as a hack) to allow all objects (including null) to accept methods like #ifPresent, #orElse, and #map? Wouldn’t making this behavior available to all variables solve many of the issues associated with making Optional<T> a wrapper class?

  • There would be no Optional class to ponder how best to use.
  • Stream terminal operations work fine.
  • There are no issues regarding whether it should be a field or not, serializable or not, etc.
  • Creation semantics between #ofNullable and #of
  • Existing frameworks that deal with beans wouldn’t need to change.
  • No need to accommodate older classes that do not return an Optional<T>.

If we go back to our earlier example, instead of this:

String value =
   Optional.ofNullable(map.get("foo"))
      .orElse("bar");

We would be able to always and safely write this:

String value = map.get("foo").orElse("bar");

No code needs to re-written to make use of optional behavior, you no longer have to complain that an Optional should be serializable, wonder where you should use a normal object or instantiate an Optional wrapper for that object, etc. And you would never wonder if you should declare a map as Map<String, Optional<String>>.

So you may wonder why the Java experts didn’t make null and java.lang.Object respond to the Optional<T> methods. Well this idea does have at least one VERY BIG issue. If your code (or any code you depended on) happened to contain a class with a method named #ifPresent, #orElse, or #map you’d be screwed – especially if any of those methods had an incompatible signature. Your code would not compile. Wow…that sucks, built-in Optional methods seemed like such a simple solution.

Is there a way around this issue? Well, some languages use an operator instead of methods for optional behavior and that could get us past part of the problem. It’s sometimes called an Elvis Operator or Null Coalescing Operator. For example, in Groovy or PHP you might write the above code like this:

String value = map.get("foo") ?: "bar";

You can think of it as shorthand for:

String value = map.get("foo") != null ? map.get(“foo”) : "bar";

That works great for #orElse, but it doesn’t give us that cool #ifPresent functionality. For example:

json.getUserName().ifPresent(dbRecord::setUserName);

With an Elvis operator you’d have to do something like this:

dbRecord.setUserName(json.getUserName() ?: dbRecord.getUserName());

As you can see the two variants are not functionally the same. #ifPresent allows us to complete forgo calling dbRecord#setUserName when json#getUserName is not present. So I suppose we’d need something more than a null coalescing operator to get rid of the Optional<T> class but perhaps for Java 8 that one operator would have been enough.

The JDK 8 team had many discussions about this, so clearly there is no perfect solution. Here are some ideas I’m sure they threw around:

  1. Make all objects (including null) respond to #ifPresent, #orElse, and #map and let the chips fall where they may. If your code or a library has a class that uses these methods then you get a compilation error. You either can’t use the latest version of Java or you have to update your code and/or libraries.
  2. Make a special message send operator for these built-in and universal methods (system messages). For example, value:ifPresent or value->ifPresent. Ew. I don’t like this idea but I still like it better than I like instantiating Optional wrapper objects.
  3. Create an Elvis operator and leave out friends like #isPresent and #map. Again, not my favorite choice but still better than instantiating Optional<T>

My preference is #1. But hey, I’m not responsible for the adoption of an entire language upgrade. I wouldn’t get the heat for any broken code. Then again, most popular libraries respond quickly to changes. If I think about it, the slight chance you may have to remove an existing #ifPresent, #orElse, or #map from a library is probably less effort than deciding what to do if a Java Bean has a getter that returns an instance of Optional<T>. And code that is mine can usually be fixed quickly. Most issues could be determined at compile time. I don’t think there are a TON of classes that have these methods out there. I did a search on GitHUB and most of the hits for these method names use the same method signature as Optional – so those might be able to stay. Is it a reckless choice? Yup. But look at the alternative. Every day more and more libraries are having to adopt and adapt to use Optional<T>s (Jackson, Gson, Jersey, Hibernate, Immutables, Lombok, Spring, etc.) and everyday users are introducing more and more fucked-up usages of Optional<T>.

So while there are no perfect solutions there are better solutions than a wrapper class with ambiguous semantics. The presence of Optional<T> behavior on an object should not be optional. Optional behavior should be implicit on every object whether it is null or not. Introducing Optional<T> into Java 8 was a simple solution to a complex problem. But it has introduced its own set problem that are far reaching and will last in perpetuity.

So how will I use Optional<T> now? Until something else comes along I will follow the advice of Brian Goetz.

Simple Basic Authentication for a Jersey JAX-RS Server

If you are working on an MVP for a start-up you are probably doing many things you have never done before. If one of those things is a REST server please make sure AT THE VERY LEAST to require that clients use HTTPS and that you add an API KEY and SECRET for your REST server using Basic Auth.

To learn the basics about basic auth go here or just google around.

For this post I assume you are using some language that can make use of Jersey JAX-RS version 2.7 or higher.

Okay, so you have written a fucking awesome REST server for your mobile apps to use. You are such a bad ass because it is also running on something like heroku. The REST server does what the client needs, the clients are basically done, so you are ready to go live with your MVP. STOP!! Stop what you are doing and ask yourself this. Have you added at least a basic layer of security? If you haven’t you will most definitely regret it later. Once you have released a client without HTTPS and Basic Auth it is hard to go back and require it. That’s because those HTTP clients will still need to be supported even after you introduce a new version that has some security. So stop what you are doing and follow these simple steps.

  1. Go to my GitHUB repository and add this code to your project.
  2. Annotate your resource classes with @BasicAuthentication
  3. Add the BasicAuthenticationFeature to your ResourceConfig.
  4. Distribute your new API KEY and API SECRET to your client to use.
  5. Have the client modify its REST calls to your API to use Basic Auth.
  6. Feel a lot better about everything in the universe.

You don’t really need to read on at this point. The unit tests show you how to annotate some resources and generate keys.  You can always add layers of user specific authentication or authorization on top of this but at least start with Basic Authentication and HTTPS.

If you are going to make your entire REST API authenticated you will only need a ContainerRequestFilter that performs the authentication. This will work in most cases but sometimes you will also have some unauthenticated resources in your API. For example, an HTML resource that you want anyone to be able to hit (not just your authenticated mobile client). For those cases I created an annotation named @BasicAuthentication. If present code will be authenticated. If not present or set to false (e.g. @BasicAuthentication(false)) then the authenticating filter will not be invoked for that resource method.

All the code you need is in GitHUB at Jester.

To use it in your server you need to generate a KEY and SECRET for your API and install the feature.

To generate the API key and secret I just use the result of calling UUID.nextRandom().toString() twice. Once for the KEY and again for the SECRET. Then I create my Jersey ResourceConfig like this:

public class JerseyResourceConfiguration extends ResourceConfig {

    private static final String API_KEY = "8694c2e7-a9c4-4646-acdb-8dbe8e8829af";
    private static final String API_SECRET = "75c4bca4-5bf9-4c91-99cc-e640ee6cded7";

    public JerseyResourceConfiguration() {
        packages("com.myapp.rest.service");
        register(new BasicAuthenticationFeature(API_KEY, API_SECRET));
        // whatever else you register
    }
}

That’s pretty much it. I keep my source and jars in a secure location so I embed the string literals into the source. You may want to take another approach to ensure the privacy of these strings. You should find lots of examples in the test directory of the GitHUB project.

Next give these strings to your client developers and tell them to make REST calls with basic auth. You can also use curl to test your API.

curl
-u "8694c2e7-a9c4-4646-acdb-8dbe8e8829af:75c4bca4-5bf9-4c91-99cc-e640ee6cded7" https://myrestapi/dosomething

I’ll try to expand on all this later but I wanted to get something up quickly for those of you who would otherwise push an unsecured REST API onto Heroku.

Don’t Create a Blob

An overly general design says to the world “We do not know who our customer is and how they will use this product.” Rather than doing the hard work of experimenting and making a choice when instead gave the customer many choices. Over generalizing makes your product both muddy and overly complicated. A friend of mine, Chris White, used to call overly general products an “Amorphous Morass of Human Insensitivity.” Have you ever used a product that was “powerful” but doing anything with it was very “complicated?” I mean so complicated that it hurt your brain? That product was an Amorphous Morass of Human Insensitivity.

Note that generalization is not bad thing it’s just that you need the right amount. A platform, programming language, or automation tool clearly needs to be more general than a photo sharing app. You will know you are crossing the line when you make simple task a little more cumbersome for a specific user in order to allow more fringe use cases that “might appeal to a larger set of users.”  You might say “While 80% of our customers use workflow A we have a significant 20% of customers that use workflow B and C.” So your answer to this issue is to make your product model ALL workflows. How cool is that? It is totally customizable. No not really cool. You just made life more difficult for the 80% that use workflow A. You could have made something very intuitive and amazing for that 80%, but instead you chose to make something that is just kind of okay for everyone.

Over generalizing can also lead to feature bloat where you have many ways to do the same thing. This leaves your user wondering which way is the appropriate way to carry out a task. Now they have to think, do I use approach A or approach B, C, or D? Don’t give them lots of ways to do the same thing, invest in finding out the most intuitive way and give them that. Give them the right way to do things.

What is my product’s special purpose?

As an old-school object oriented programmer I always wanted to make my architectures as generalized as possible. If I was building an application for categorizing music I could see that I could build it in a way that allowed the application to categorize pretty much anything. Once you fall down this rabbit hole you start doing all kinds of crazy things. For example, you might make the workflow around categorizing music just a little more cumbersome so that it could also be applied to images or documents or “Hey, man” pretty much anything else you might want to categorize! How cool is that?!?” Not really that cool at all. “This architecture is so cool you can pretty much do anything you want to with it!” No, the architecture is sad. Very sad for those customers who wanted an innovative and intuitive way to categorize music.

Product designers and developers do this when they do not understand the unique purpose of the application they are creating. They don’t have 100% certainty that they have a good value and how to deliver that value. So they start adding in a bunch of unnecessary and overly general stuff. Creating few very well defined personae and context scenarios solves this problem. The culture should be that if a use case does not serve an agreed upon persona then it is not added and if a use case is not present it is not considered. If you have a compelling new use case that does not have a persona then you have a business decision to make. You are making a decision to appeal to a new customer or user. You are essentially going into a new market. Don’t do everything you “could” do. Find your special purpose but don’t be a feature slut.

You have to make choices

Let’s say that you have a systems management tool. You know that your users need a better way to manage their nodes. So you hypothesize that you should have a node tagging feature. Great. So far so good. While validating this hypothesis you find that one or two high profile customers really prefer hierarchical grouping for organizing their nodes. What should you do? If your answer is to add the ability to do both then you have answered incorrectly. Answering YES is a step towards creating an Amorphous Morass of Human Insensitivity. New users will wonder, should I use groups or tags? Every feature that targets nodes will need to be done with both “grouping” and “tagging” features. Not being able to say no is a lack of commitment that impacts your customers as well as your development effort. Now you’ve confused your primary customer and you have twice as much code for your developers to maintain. Forget about grouping and deliver a kick-ass tagging feature.

Product Managers love to add features when they don’t want to do the hard work of finding a specific and powerful value for their product. It’s much easier to create a backlog with a bunch of customer requests for “additional features” than it is to figure out the single strategic value and then put all the wood behind that arrow. In music it’s easier to riff on a theme than it is to create a new theme from nothing. But adding too many riffs or layers on a theme will eventually kill it. If you keeping adding features to your product you will eventually ruin it.

You Aren’t Gonna Need It and MVP

YAGNI (You Aren’t Gonna Need It) is a programming concept from XP that is just as important for Product Managers especially when trying to define your Minimal Viable Product. The more features you add without validation the more legacy features you will introduce into your product. The more features you add the longer it will take to release. The more features you add the more muddy you will make your value proposition. Pair down your backlog to just essential features and then take even more features out.

The famous example here is the iPhone. Remember that the first version of the iPhone did not even support cut and paste. As a product manager or owner, how many of you would have made that decision? I think very few. But as a result the iPhone came out quickly and THEN due to customer feedback the feature was added. Determine the features you need for an MVP and then take one away. Not having the so-called essential feature of cut and paste did not prevent the iPhone from being successful.

The first Kodak camera was created for one reason, to sell more Kodak film. However this camera was so simple that it caught on like crazy and transformed Kodak’s business. It was nothing compared to other cameras available at that time. All you could do was point it at something and click a picture. Then you sent the film into Kodak and they developed it for you. There was no F-stop, zoom, flash, etc. Sometimes fewer features give you a more powerful value. Always keep Instagram in mind. This product with very few bells and whistles beyond its primary value proposition sold for a freakin’ billion dollars. Holy shit! Some would even argue that it didn’t have enough features for its value proposition. Find out what features you absolutely need for an MVP and then take some away.

When you find yourself thinking that a product launch was not successful because some essential feature was missing remember the iPhone, Kodak camera, and Instagram. You are really just trying to make yourself feel better for not finding your product’s special purpose. It was not the lack of features that kept your product from achieving revenue goals. It was that you did not find a winning value proposition. Value, not features.

When you don’t have a clear value you need all the features in the world. When you have a well defined value you hardly need any features. If you find yourself adding a bunch of essential features consider if you really have figured out your value.

10 or so Signs that R&D is Sick

A lot goes on in a business but this is about R&D. You should read this even if you are not part of an R&D organization. Sometimes someone outside of the R&D organization needs to spot the insanity because those in the shit in R&D can become too subjective.

Whether you are just starting out or growing an already successful company there are fewer things more destructive to company growth than a sick R&D organization. All parts of your business are important and clearly you need a great idea that speaks to a market but in this post I am going to focus on your R&D organization.

Over the years I’ve observed some traits that may indicate your R&D organization needs a health check. Some of these rear their head right after you’ve had some success and feel you need to “improve” and “grow” your R&D organization. I’m unique, so some of these may make you bristle or you may think I’m insane. Fine, but I know what works and I know what doesn’t work. These are in no particular order. I’ll try to expand each one of these into their own blog post.

1. Process over Productivity

Concentrate on producing not on process. Managers will often work on process rather than address the real problems. I don’t know why, they just do. Working on process makes them feel more important and like they are doing stuff. Process will not make up for a lack of talent, direction, customer validation, and vision. It just allows the VP of R&D to go to executive staff with velocity charts that show R&D is kicking ass even if it is not shipping product and not making customers happy. Have you spent time in meetings talking about the true meaning of a story card and what it should contain? Get a life and start producing and spend less time on silly conversations that probably don’t matter that much. Who cares if story cards vary a little. Optimize the right things. Focus on Product.

2. Technology Centric Architecture

Architecture needs to be customer centric not technology centric. If your engineers are creating overwrought architectures then they need to spend more time with customers and sales. Spend more time coding and designing quick features you can test with customers and less time on a really cool architecture. Perfect features after they have been proven to be valuable. “Build it right the first time” is BS.

3. Managers That Don’t Code Making Technology Decisions

You need leaders not managers. If you have managers that are not practitioners they should not make technology decisions. If they want to make technology decisions then they should also be responsible for doing some coding, designing, testing, or whatever area it is that they manage. If you need a bunch of managers then you may really have a direction and talent problem. Compensate your top individual contributors more than your managers. This will keep them from thinking that managers are more valued and that managing is the only career path in the company.

4. Having a “No Heroes” Attitude

Despite what you’ve heard sports teams and rock bands do have heroes. It’s okay if you do too. People try to emulate heroes and having a hero around raises the bar for everyone else. It’s okay if you have a super star that is peculiar or can be a pain in the ass. Just remember what a super star really is. A super star is NOT someone who can bully other people or quote Knuth. A super star is someone who consistently creates superior customer value quickly. I’m not really a huge Van Halen fan but ask yourself this. Which Van Halen was better? The one with crazy David Lee Roth or the one with the very manageable Sammy Hagar?

5. A Separate Bug Fixing Team

This one always bothers me. At a certain point a company feels that it is too big to have the same people that code features fix bugs in those features. The result is a development team that is out of touch and is not accountable for their mistakes. Trust me, you won’t destroy velocity if the developer that wrote a feature has to fix a bug in it later on or deal with a customer escalation. What you will have is a developer that understand the customer pain they created and be more careful to write solid and usable features in the future.

6. Customer Escalations Stop Being a Top Priority aka Big Strategy Over Customer Satisfaction

This is kind of like the above and it usually comes from the VP or management. At some point a company grows to the point where they believe they have bigger fish to fry than customer satisfaction. If you are going too slow to keep customers satisfied and work on new strategy at the same time then your organization is sick. Find out what’s really wrong in the organization and fix it. There is no better advertisement for a product than word of mouth. There is nothing more important than making sure a customer’s deployment is working. This does not mean creating one off features for customers. Don’t do that! That will give you a directionless product. Know when to say no to one-off customer requests. But do make sure your customers are happy with your product’s intended use. Always make these a priority even if you are a big company with lots of sales. Treat each customer like you did when you were trying to make your first million dollars in annual sales. Remember those days? Remember how horrified you were if a customer ran into a bug that stopped them from using the product? As an engineer you should be embarrassed when a customer hits a show stopper issue. You should not be able to sleep until it is resolved. If you don’t care that you fucked up a customers week then you are either out of touch or a sociopath.

7. The UX Team is Marginalized

Do not marginalize the user experience team. In the best case scenario UX should not even report to R&D but are a peer of R&D and Product Management. UX should be part of the holy trinity that leads a product line — Product Manager, Technical Lead/Architect, and UX. UX are not visual designers. They do not exist primarily to make the product pretty or sexy. They are essential in making sure the product provides value to customers. They run experiments, validate results, and make sure the product is useable and on point.

8. The Engineering Team has no Customer Contact

Make sure your engineers get out of the building. They need to understand the product and its customers not just the code. Once the developers loose contact with the customer things will soon go catawampus. You will have a technology centric architecture that is more than is required. Simple ways to keep this from happening is to send engineers on POC, Professional Services Engagements, on-site escalation resolution, or even sales calls. You could even have engineers rotate on dealing with a technical support issue or two every once in a while. The idea is to have each engineer know how the product is used, what business it is that we are in, and understand the customer pain.

9. The Team is Resistant to Feature Changes

This one is usually a relic from an older style of doing business. You have this when you see R&D asking for detailed product requirements or marketing requirements documents or complaining that the backlog keeps changing. In today’s world businesses (especially start ups) need to change sometimes even day to day based on customer feedback and market changes. Having small units of work that fit into cycles helps with this. If you have engineers complaining that they can’t fit a feature into a cycle then you probably have an over engineered solution. Get features out there quickly, validate them with customers before and after release, and then perfect the ones that have traction. If you are in sprint 3 and the product manager changes the features you were going to do in sprint 4 then the product manager is doing her job. If you are organized so that PM, Architect, and UX form a trinity then they should all understand why the backlog changed.

10. Tension Between R&D and Product Management

The leadership in R&D and the leadership in Product Management should be pals. If they are not pals you will soon see a rift between everyone in R&D and product management. R&D will blame PM for all the ills in the world and product will not get validated and not get delivered because of in fighting and endless debate. An adversarial relationship between R&D and PM could kill a company. R&D needs to believe in what’s it building and PM needs R&D’s support. If you are organized correctly, meaning that each line is lead by a Product Manager, a Technical Lead, and a UX Lead and they are all held accountable then this should not happen. They are team one. In other words, a technical leads primary team should be the Product Manager and UX lead she works with and not the engineering team spread across multiple products or components. Don’t fuck with PM by asking them to specify every last detail of a story. That is old school. R&D, UX, and PM should do that together.

Conclusion

I guess all of these really boil down to just a few points. R&D must be connected with the customer, work in concert with Product Management and UX, experiment with minimal versions of features first, and value productivity over process.

 

 

When you’re green you grow when you’re ripe you rot.

“Anyone who has never made a mistake has never tried anything new.” – Albert Einstein

Okay, so I’m going to start my blog with this video. Anyone who knows me knows how much I like this talk by Derek Sivers on failure. That he is from Portland is just an added bonus.

Probably my favorite part of this presentation is where he talks about Picasso being video taped while in the process of painting. The painting goes through many revisions, at some points Picasso even seems unsure of himself. Ideas are crossed out and redone until he gets to his finished product. He made many mistakes to get to his completed painting. The big take away here is that you cannot compare your internal self to the external self of someone else. Think about it. You know everything about you, you know all of your insecurities, the circuitous route you take to finally arrive at a decision, etc. For someone else all you know is the final product the present – you do not see their internal process. So do not compare your inside to the outside of someone else. You will always come out short.

Why You Need to Fail – by Derek Sivers from Derek Sivers on Vimeo.