Simulating microservices using F# and Suave.io

I have this small hobby project called Overseer. It is a tool used to debug and troubleshoot microservices. I was planning to use it in production, but because that is the only prototype. I decided to use it with simulated services. This way it will be a lot easier to create many different ‘test’ scenarios.

The current solution for spawning those ‘tests services’ is implemented in Suave.io and F#. Why these technologies? I just like F# and Suave.io seems like a nice decent framework. Full code If you can, please review this code. I am not a F# expert and would love to learn how to make it better.

Test service graph definition

I want to iterate fast and create different networks of services. I need a simple way of defining the structure of the network. At the moment, I am using an array of tuples with a service name and collection of all the services it depends on. It looks like this.

This array of tuples is changed to Suave web ‘servers’ with one endpoint.

The result is a list of Tasks running suave exposing one simple endpoint.

Endpoint list

To create a list of the endpoints, I am using fold function with an accumulator to increment port. Fold is a function that I tend to overuse too much. But in this case, it feels ok. Is there any other way to iterate through something with a maintnance of external state?

Suave config

It was a bit tricky as the documentation is not that good. After many try and errors and SO searching, I was able to create simple endpoints. All it does is returning JSON metadata describing ‘server’ (status, dependencies).

Result

All the dependencies, links to them, ports are here. This data is then used to create a graph app id d3.js, example.

alt text

This is still in the making. D3.js is really powerful but quite tricky to use. I am mostly experimenting and there are a lot of ‘wow’ moments. Kind of feels like winapi programming a long time ago.

The first test was done with 11 services and it ‘works’. Wonder how it will cope with 100+ services. That is a topic for a different blog post.

University degree and IT

University degree importance in IT is a big topic of many discussion around the internet, I also received a question like that.

Q: Is University degree really that important for future career in IT sector. Wouldn’t it be better to start working as soon as possible and sacrifice study time for more work experience. I am a student now, and I do feel now that studying is limiting my potential and growth. The things that, I am doing now in my work are completely unrelated to my courses on uni.

Unfortunately there is no simple answer. There is no ‘Yes’ / ‘No’ here. Both more study vs more work experience have their own pros and cons. In this answer I will focus on ‘Why I think university time is important’.

My Experience was that. I am a proud alumni of Wroclaw University Of Technology. I studied Electronic and Telecommunication. It was lousy related to Computer Science but I was on Applied Computer Engineering faculty which had a lot of computer science courses. I spent a lot of time in the really low level world of computers. Writing code in ASM, C, C++, learning about electronic circuits boards, designing and soldering stuff. Even if this was not really full Computer Science course. I still feel that this knowledge has positively influenced me and I easily picked up all the skills required for my future roles. With current knowledge and experience, would I enroll again ? Definitely!

For me studying was a time of opportunity and moment to learn many different things, try different technologies and gain a lot of theoretical knowledge. It is really important to try and experiment with everything as much as possible. I am not talking here about languages or frameworks but also different areas of computer science. Like i mentioned before, I was doing ASM programming, C Programming, OOP programming, database programming, image recognition algorithms implementations, bird flocks simulations, genetic algorithms things etc plus i spent some time soldering and programming micro controllers. One big mish mash of things. Having such a broad experience enables me to have a nice view on many different things. It allows me to learn new things fast. It is difficult to focus on those areas and this type of learning when you are working.

As for first work experience. I started working on my 3rd year. I was 22 when I joined my first company it felt like all the knowledge and courses on university were unrelated to work. My boss demanded practical skills not some theoretical mumbo jumbo. I was lucky enough to start my career in a Salesforce project ( yeah! ). It was really hard for me to find a link between Uni / Workplace demands. Back then however, I didn’t understand how important was that theoretical stuff that I was learning and how it has increased my brain capacity. It was normal for me to assume that Uni time was a waste and there is no way I am gonna use all this knowledge in the future. Ohh how wrong I was. At the start of career you can’t really assume anything. You don’t know where you will go and what you will work on. Learning AI programming 5 years ago seemed like a waste of time, but look what happened with all the machine learning and big data stuff becoming more and more popular. At that time, with lack of experience and knowledge about the business, it was impossible for me to predict those shifts.

Studying is important as it lets you build a base for future growth. You cant build a house without laying the foundations first. You have to start from the bottom. By learning and trying many different concepts you are training your brain to learn faster and you are creating connection in your brain that will enhance your learning capacity. Learning is something that you need to train. Also learning about new concepts is a lot easier if you already know many other things. That is why I encourage everyone to try as much as possible on UNI. Don’t focus on one platform and don’t focus on gaining marketable skills. The time will come for that later. Try to reach out your professors and discuss some technologies, frameworks concepts that you would want to try out. All of this will start repaying after 4-5 years in the market when you will realize that developer job is not code but find solutions and new ways to generate value for the business. You can do some work, just try to not stress out about it and don’t let it become the main focus of your Uni time.

TL;DR; You don’t need degree in IT but University can be a huge catalyst for you career and growth, if you choose to use this time wisely. Can you achieve the same growth outside of university ? It depends, if you will find a group of like minded people that will help you learn then yes. Because studying is mostly about people you meet and make connections with. University environment shaped me as a mature engineer and also mature person.

Take care of your test code

As developers we tend to write complicated unit tests that are hard to read and maintain. Maybe it comes from the feeling that test code is not a proper code ? There is some magic in writing ‘proper’ unit tests. Using word proper might not be even suitable here because how do you define ‘proper’. It is the same problem as with the definition of ‘unit, everyone has his own definition that depends on the context.

Code below feels like not ‘proper’

This one was found on production (slight changes applied). I don’t really like the function name to start with.

IfUserIsAuthenticated_and_Admin_return_View()
It looks ok but what does it really say ? It does something for scenario with authenticated user, probably an admin user and it returns some View.

Code

  • mocks some kind of authentication service that uses HttpContext
  • sut is a controller with a nice list of dependencies
  • we do execute sut with edit and a list of variables
  • and some view is returned in the process

Does this test looks good ? Does it reads really well ? I don’t think so. The mocking logic, as usual, is making it complicated. Creation of the sut is also full of surprises with its number of dependencies.

Improvement

EditView_is_accessible_to_authorized_user_of_admin_type()
Not perfect but better, clear message on what is happening here.

Code

  • Sut creation is hidden behind the helper class with an ‘interface’ exposing only important bits. I need to control if the user is authorized and what kind of type he is to perform this test
  • Edit call is very simple with variables that are not affecting test are hidden behind variable ‘_’. This is usefull technique to hide non important parts of the code
  • Check for the access is hidden behind custom assertion. We don’t need to expose details here.

Simple changes yet making a huge difference.

Daj sie poznac – Overseer

Polish .NET community and Maciej Aniserowicz has started another edition of “Daj sie poznac” event ( translation: something along “Show yourself!” ).

The last one was probably in 2010. The community was completely different back then. Not that many people writing about tech stuff. Comparing to current number of people that joined the party ( 100+ ) a lot has changed since 2010.

The rules of the contest are very simple.

  • Have a hobby project
  • Write about the project and progress
  • Contribute to the community
  • Have fun

More details (It is in Polish)

I was working on small hobby thing and this will be another way to encourage me to do stuff and code / write.

Hobby Project – Overseer
A simple microservices monitoring app that will show the graph of all the services with their dependencies. A way to check the status, traffic and maybe debug requests later on ( that would be cool ). This will be useful tool in my company and a great way to learn and do something interesting.

Things to do:

  • Simulated services. I need a way to create multiple services in controlled environment. Planning to create this using F# with Suave. Might add docker later.
  • Front-end UI. React App which for data presentation D3.js for graph rendering.
  • System to gather the data for the React App. Potentialy will use the Akka.Net.

Github link for the project

I am also going to stream coding sessions live using livecoding.tv platform.

BuildStuff 2015

Finally, I was on Build Stuff, for the first time. I heard a lot of good things about it and my expectation were pretty high. This is mostly due to my friend @sawiczpawel and his recommendation ( Thank you Pawel! ). In short, It was a good conference, I enjoyed it, learned a couple of new things and meet brilliant people.

The best

Let’s start with session and speakers who I think did the best job.

Motiejus Jakstys (@mo_kelione) - Unikernels and the future of secure cloud computing.

Most interesting talk of whole BuildStuff. The unikernels concept is something I have to dig into more. The whole idea sounds really interesting, transforming your apps into mini unix kernels. This could be a counter-weight to current docker hype. I was kind of angry that being part of MS stack is causing me to miss out some of the cool unix world things.

Venkat Subramaniam (@venkat_s) – Let’s get Lazy

Venkat presentation skills are just excellent. He might be talking little bit too fast but I was able to keep up. This presentation was about lazy loading examples in different languages. Nothing really exceptional. This talk was probably intended for JAVA coding group. Still seeing how Venkat does a talk on live is a good experience. My colleagues told me that his Javascript talk was also excellent.

Ben Hall (@ben_hall ) – Real World Experience Report on Running Docker

It was good to see someone who uses docker on prod. I am still trying to get into docker, being mostly on MS stack doesn’t make things easier. This talks was mostly a case study by Ben. I found couple of nice tips and interesting ideas, nothing revolutionary. I was really impressed by Ben’s presentation skills. The flow of information was perfect, slides organised in correct order. This was technically ( speaking, information organization ) the best talk of the whole conference. Some of the speakers could learn a lot from this talk.

Uncle Bob / Robert Martin (@unclebobmartin )- The Last Programming Language

The Keynote was great and organised but the important bit was was the message. We need to nourish our engineering culture and fight for it. With huge influx of new developers each year it is the duty of more experienced devs to show those new people how to build stuff correctly. This presentation got me thinking a lot which is the best sign of a good topic and delivery.

The good

Networking / People
My networking skills are, I don’t want to use a harsh word here, let say I am novice in this. Still I managed to finally met @orientman, @ptrstpp950, @liveweird, @wmekal and a lot of other people. Had good time and even got 6th on a BuildStuff poker tournament ( turns out poker play is not the best place to meet people, everyone is so concentrated on the game that there is no time to talk ). It felt like Build Stuff is the conference that attracts the kind of people like me. The atmosphere was really great, I could feel that I am surrounded by like-minded people.

Organization
It wasn’t perfect, there was some confusion with the time / place of the talks, but overall it was good. The venue was excellent, food was great. The stands with various games, neat idea and arcade stand in the chill out lounge was awesome. I also liked the draw your own caricature idea ( I won’t show you mine because it is horrible, yeah really it is bad).

The bad

There is only one complaint and something that I cant really understand.Some of the speakers and their presentation were really bad. I don’t know how this happened but it looked like lack of preparation. It is quite surprising as some of those speakers are quite good when they want to. Ehh, it was confusing, they lost it somewhere. Sure it can happen but, I don’t know I really have mixed feelings. I was really hugely disappointed on couple of talks.

Sebastian Gebski described nicely some of my feelings on his blog

Overall

Build Stuff is a good place to go, full of energy and passionate people. I recharged my dev batteries plus learned a lot of stuff. I will be there next year, for sure. Despite disappointment about couple of presentation, overall the speakers did a great job of delivering their thoughts. I came to Build Stuff to learn and meet people, also hoping that I might get small kick in my passionate butt to start doing more stuff, I Got It!

From procedural to functional – an example in F#

In this blog post, I show little code review of a code sent to me by friend. The code is quite simple example on how to read XML file using XmlProvider. My friend’s biggest concern was that his code is still procedural and not ‘functional’. I was asked if this code can be refactored to be more functional, whatever it means.

First of all, the code that I got, is fine and most importantly it works. Even when you are using F# to solve your problems, you don’t really have to make your code super functional. That’s the beauty of this language. You can still use state-full and mutable approach to implement solution. You don’t have to make this code functional just for the sake of it.

Said that, I will still try to make the code better and in some ways more ‘functional’ to show some awesome F# features and approaches that do make the difference. My main goal here is to to remove loops and mutability.

XML structure

The XML is a representing a Collection of ‘rows’ with parameters specified as ‘cols’. Each ‘Col’ has a param name and value. Some of the params are missing value.

Original Code

  • Record type defined with all the values as options. Column values are optional thus all the properties are declared as options. This allows the usage of ‘None’ value.
  • The data is being read using FSharp.Data and XmlProvider
  • Collection of records is created by using Sequence and yield operator.
  • The core mapping code from XML to record is in the loop.
  • By default all the values, that hold col value, are set to None and then if one of them exists we do set the proper value
  • I am not sure why ‘ref’ has been used here. It can be changed to just mutable ‘let binding’. More on ref
  • Because ‘ref’ is used, there has to be ‘!’ used to get the data. Thus this weird syntax of ‘!cy’ to extract the value from ‘ref’.

So where to start with some changes ?

With operator and stateless record change

I start with a simple, yet a nice change, that makes the code cleaner.

Instead of extracting all the values and then creating a record instance at the end, I use ‘with’ operator and compose the Record on the fly.

In order to do this, I need to define a function to create a default record type with all the values set to ‘None’ ( option type ). Thanks to this, I can remove ref’ and weird initialization of temp values.

Then, I introduce a function that maps the ‘col’ and its value to proper field in the Record

This function takes a name of the ‘col’ ( parameter ) , a number or a string and the existing record. The with operator creates a new Record based on existing one with specified field added to it.

Now, I can write code like this.

This code creates newRecord and then adds voivoship value to it. All the existing fields are still set to ‘None’.

From loop to recursion

In the original code there is a loop used to change the state of the object. This is a perfect approach in procedural statefull approach. You just add new fields to the object. However when, stateless approach is needed, loop won’t fit it due to its dependence on external state change. Loop doesn’t return anything and it can only change existing state out of its scope. If you want to change something in the loop it has to be mutable.

If you want to loop some collection and change something without mutable value then you might use recursion. The good thing about it is that recursion is a function that has to return something and you can return new state. There is no need for mutable values.

To change the loop from original code to recursion, I need to define a recursive function.

  • rec means that this function is recursive
  • this function takes an existing record and list of columns
  • it loops through the list of columns using match
  • ‘head::tail’ is a list deconstruction convention using cons operator - ‘head’ will be the first element while ‘tail’ is the rest of the collection
  • to do recursive call, I invoke the same function passing record created by addCol function ( introduced earlier in the post ) and rest of the collection ( ‘tail’ )
  • if the list is empty i return the record
  • this is a very common pattern in functional programming

To bind everything together, there is a function that initiates the recursion

  • I do declare a function iterateCols
  • and then invoke it with initial parameters ( an empty record ) and ( list of columns )

Sequence now returns a collection of Rows with values from Columns

That’s all. Full code is available here.

Recursion vs Loop

When you look at the original code and new code, it might look less readable. I do find recursion more readable now, but I got used to it. Normal for loop might be more readable in some cases and you have to always ask a questions like.

  • Do, I need stateless code in here ?
  •  What are the benefits of making this code more ‘pure-functional’

Like everything in our field, there is no silver bullet and we have to be pragmatic.

Feel free to leave a comments with your opinions about this code. I would love to hear more suggestion and ideas on how to approach problems like this.

I want to be programmer where to start ?

I have received an email from a 15th year old student from US asking the question on where to start the coding / programming adventure. I want to share with you questions and my answers.

Q: “I am currently 15 years old and a freshman at University City High School. I have an interest in becoming a software developer. I was curious to know if I should wait until I am older to start learning different programming languages. I am good with a computer, and have a B in geometry right now so I would say I have different strengths and weaknesses in math. Just to add, school is stressful for me so there are nights where I wouldn’t have time to practice coding and I play travel hockey from August to February, and that can make me stay up late at night. If you have any suggestions on software developing that can be user friendly for a high schooler please tell me! And let me know if 15 sounds right to start, or if I should wait.”

A: It is awesome that in age of 15th you are considering career in tech. When I was your age, I was only interested in playing football and video games :) . I had however started playing around with a Pascal language ( something really old ) and this is how it started. It was my father that gave me this first spark.

You don’t have to wait to start learning languages, you can do it now, but don’t stress over it, think about it more as a fun activity, not as work. You still have a lot of time to learn and in your teenage years you need to have fun. I would suggest you to start your adventure with JavaScript. It is a fairly simple yet powerful language that you can quickly do really fun things. The most important thing in learning at this stage is to quickly see some useful results. That’s how you will build your own internal motivation to learn more. This is very important for programmers.

When learning programming you have to remember that at start it might be hard plus you will not see the results immediately, but after some investment awesome things will come. Software dev is one of those rare career paths that you can have fun in your work place. I love comming to my work every day plus I am getting paid a decent amount of money. It is really great.
Don’t worry about time, you are still 15th, treat coding practices as something you can do but it shouldn’t be a priority for you now. I started programming for real when I was 22, I know people that started this path in their 30′s. So it is not like you have to start now and work hard when you are 15th to have an awesome career. There is still time.

When it comes to learning resources.
You could start with:
https://codecombat.com/ – this one will teach you the programming basics + it is fun :)
http://www.jsdares.com/ – quite nice course also with games
http://www.codecademy.com/en/tracks/javascript - examples not related with games

Q: “I saw a video about a software developer saying how in his opinion, the good coders are the ones that love doing it and want to make really good code.”

A: Yes it is cool to be a great coder / programmer but the reality is that most of the positions require average programmers and it is fine to be average. Sure being a great coder / developer gives you huge advantage but don’t stress over it now. With time you will know if you are good at it or not, then you can decide to try something else.

Q: “According to the U.S. Bureau of Labor Statistics, software developers who work on systems software will have an employment growth of 20% from 2012-2022. For developers working on applications will have a 23% growth during the same time period.

A: Yes the demand for developers is huge. It is a huge gap and it looks like it will be like that for the foreseeable future. This demand is driving the salary up, plus you are treated nicely by employees. It is really a nice market. This profession is also safe from all the robotics advancement that might replace a lot of other crafts. There is a risk that advancements in AI might make this profession obsolete  in the future but then all creative work will become obsolete.

Q: “Are there any books you can recommend that can help me with coding or just about software developing in general?”

A: There are two book that I will recommend for starter. Head First Programming - yes this book looks silly but it is serious. It has a nice approach to learning, suitable for everyone and then you can also try Head First JavaScript Programming

Q: “From your experience working as a software developer, how much of your day is just coding and the other part of your day attending meetings and such?”

A: When it comes to how my work day looks like. It depends. You have to know that software developers are doing many different things, yes when you are junior developer you spend most of the time coding. That’s how everyone started. With time and experience this changes and you are getting more diverse tasks. Some of us are going into more technical roles, some are getting into management, some are leaving the field to pursue completely different career. When you are Junior you are doing simple tasks and you are dependent on your colleagues. In Mid Role you are not dependent on your colleagues and you are also mostly coding. In Senior roles you have to support your colleagues by mentoring them, helping them plus you are also responsible for various decisions and you take a lot of responsibility for the product. Then you can go to Leading roles which require more soft skills.

When it comes to my day, I spend 70% of time doing technical things and 30% of time doing other things like – task management, helping team, helping other teams, doing presentations, attending meetings etc. Btw, meetings are not bad if those are handled nicely.

Q: “Do you have to be a very creative person?”

A: Yes you have to be fairly creative in this field but creativity is a skill that you can train and acquire.

Q: “You said in your blog that teamwork is key. I play hockey so the only way you can get the job done is by teamwork, no solo work will get your team to the top. I am very good with working with others because of this. So could it happen that the co workers that developers work with can try to do solo work, and not much teamwork? If this were to happen, should a person in a higher position be notified?”

http://www.mfranc.com/career/software-developer-career-hard-truths/

A: As for the teamwork. Team Sports are a great way to learn on how to act as a team. Keep on it. When I was your age I was playing football and basketball. There are a lot of programmers that do like solo work but this is limiting them. Software development is not a solo work you need a team and good teams can do much more. You will of course meet solo players that do like to work alone, sure you might go to superiors but your duty as a colleague and good team player is trying to involve those solo players to play with the team :) It is possible with help of your team leader, he should be notified that there are some problems, but politely without aggressiveness.

TDD / Unit Testing big list of learning resources from basics to advanced topics

One of my fellow dev friends asked about a set of links, books, screen-casts related to TDD / Unit Testing. He wants to expand his knowledge. Instead of sending him a private message, I thought that it would be great to just create a blog post with all the resources, I used in the past to learn.

Basics

Presentation - Video: Unit Testing and TDD – Why You Should Care and How to Make It Happen By Roy Osherove 

If you are completely new to those concepts then this presentation will be a nice quick start.

Book – “The Art Of Unit Testing” by Roy Osherove

Start here if you want to get into TDD and Unit Testing. Where to start then ? I started with Roy Osherove book. It might be outdated know but I think this one is a nice and easy quick start guide on how to get to TDD and Unit Testing. 

Book – “Professional Test Driven Development with C#”  by James Bender and Jeff McWherter 

Another good book to kick-start the learning process. It’s a full course starting from why TDD might help you and then taking you through the journey from TDD beginner.

Blog Post - Unit Test code with static method Console.Write By Myself 

If you ever asked a question on how to unit test static methods then my post shows one good approach to it.

Blog Post - Good unit test – One Assert By Myself

An explanation why one assert per unit of logic is the way to go.

Blog Post Series – String Calculator Kata one two three four by Myself :)

Kata exercise step by step with a commentary.

Blog Post Series – Chess TDD Kata By  

An awesome Kata with great commentary

Intermediate

Resources for developers that started using TDD practice and want to expand their knowledge.

Book – “Test Driven Development: By Example” by Kent Beck 

Yes maybe it would be best to start with Kent’s book, but I think it is more suitable for developers that tried TDD, did some smaller projects with it. In order to get most of Kent’s book, I think that you need prior experience. Kent’s book is like the bible that you will get back to from time to time to get new “A-HA” moments. It’s is really worth revisiting it wit more practical experience.

Pluralisight – Outside In TDD by Mark Seemann

This course is a great way to learn about one approach to TDD. Mark Seemann is probably known to all the .NET developers thanks to his blog and DI in .NET book.

Discussions – Is TDD Dead Series 

There was a huge debate about TDD and its future / relevance. This series of discussions is treasure trove of knowledge.

Presentation –  TDD, where did it all go wrong by Ian Cooper  

If you have watched the TDD discussions then you also can’t miss Ian Cooper talk.

Course / Screen cast - TDD Pairing Sessions – Building a Go Game by Roy Osherove

I haven’t checked this course yet, but its Roy and 8hrs of pair programming awesomnes!

Blog Post –  ‘Classic’ or the ‘London school’ of TDD by Jason Gorman

Explanation of two different approaches to TDD

Blog Post - Why learning TDD is hard, and what to do about it by David Tchepak

Blog Post - The TDD That Can be Spoken Is Not the Eternal TDD by Charles Hoffman

Blog Post – TDD is a path by Steve Jackson 

Blog Post - The Failures of “Intro to TDD” by  Justin Searls

Insights into why learning TDD is not that easy and needs a lot of practice and try / error approach.

Advanced

Topics that might be going out of TDD and sometimes into more philosophical issues.

Book – Growing Object-Oriented Software, Guided by Tests by Steve Freeman  and Nat Pryce

With some proven war stories it is time to start thinking how Unit Tests and TDD fits into code design.

 Pluralsight – Advanced Unit Testing by Mark Seemann

Another great course by Mark with more advanced topics.

 Pluralsight – Introduction to Property-based Testing with F# by Mark Seemann 

Property-based Testing is still a novelty for me, it is a topic worth exploring as it gets more traction currently.

Book – XUnit Test Patterns by Gerard Meszaros

If Kent’s Beck book is a Bible then Gerard Meszaros book is a new Testament. Yes it has XUnit in its name but it really doesn’t matter. Topics covered here are adaptable to all the technologies plus you need to read this book if you want to get a basic vocabulary around TDD and Unit Testing.

Blog Post – If you are not doing TDD … by Szymon Pobiega

Not doing TDD is not something bad, TDD is just another tool in your belt.

Blog Post -The Pragmatics of TDD by Uncle Bob

Another great post explaining that there is a lot of pragmatism in TDD

Blog Post – TDD Anti-Patterns by James Carr

Funny but knowledgeable list of anti patterns in TDD

Functional Flavour

If you are into functional programming then this might be also of interest to you.

Pluralsight – F# Unit Testing by Mark Seemann

Pluralsight – Test-driven Development with F# by Mark Seemann

Presentation - Look, No Mocks! Functional TDD with F# by Mark Seemann