{"rowid": 257, "title": "The (Switch)-Case for State Machines in User Interfaces", "contents": "You\u2019re tasked with creating a login form. Email, password, submit button, done.\n\u201cThis will be easy,\u201d you think to yourself.\nLogin form by Selecto\nYou\u2019ve made similar forms many times in the past; it\u2019s essentially muscle memory at this point. You\u2019re working closely with a designer, who gives you a beautiful, detailed mockup of a login form. Sure, you\u2019ll have to translate the pixels to meaningful, responsive CSS values, but that\u2019s the least of your problems.\nAs you\u2019re writing up the HTML structure and CSS layout and styles for this form, you realize that you don\u2019t know what the successful \u201clogged in\u201d page looks like. You remind the designer, who readily gives it to you. But then you start thinking more and more about how the login form is supposed to work.\n\nWhat if login fails? Where do those errors show up?\nShould we show errors differently if the user forgot to enter their email, or password, or both?\nOr should the submit button be disabled?\nShould we validate the email field?\nWhen should we show validation errors \u2013 as they\u2019re typing their email, or when they move to the password field, or when they click submit? (Note: many, many login forms are guilty of this.)\nWhen should the errors disappear?\nWhat do we show during the login process? Some loading spinner?\nWhat if loading takes too long, or a server error occurs?\n\nMany more questions come up, and you (and your designer) are understandably frustrated. The lack of upfront specification opens the door to scope creep, which readily finds itself at home in all the unexplored edge cases.\nModeling Behavior\nDescribing all the possible user flows and business logic of an application can become tricky. Ironically, user stories might not tell the whole story \u2013 they often leave out potential edge-cases or small yet important bits of information.\nHowever, one important (and very old) mathematical model of computation can be used for describing the behavior and all possible states of a user interface: the finite state machine.\nThe general idea, as it applies to user interfaces, is that all of our applications can be described (at some level of abstraction) as being in one, and only one, of a finite number of states at any given time. For example, we can describe our login form above in these states:\n\nstart - not submitted yet\nloading - submitted and logging in\nsuccess - successfully logged in\nerror - login failed\n\nAdditionally, we can describe an application as accepting a finite number of events \u2013 that is, all the possible events that can be \u201csent\u201d to the application, either from the user or some other external entity:\n\nSUBMIT - pressing the submit button\nRESOLVE - the server responds, indicating that login is successful\nREJECT - the server responds, indicating that login failed\n\nThen, we can combine these states and events to describe the transitions between them. That is, when the application is in one state, an an event occurs, we can specify what the next state should be:\n\nFrom the start state, when the SUBMIT event occurs, the app should be in the loading state.\nFrom the loading state, when the RESOLVE event occurs, login succeeded and the app should be in the success state.\nIf login fails from the loading state (i.e., when the REJECT event occurs), the app should be in the error state.\nFrom the error state, the user should be able to retry login: when the SUBMIT event occurs here, the app should go to the loading state.\nOtherwise, if any other event occurs, don\u2019t do anything and stay in the same state.\n\nThat\u2019s a pretty thorough description, similar to a user story! It\u2019s also a bit more symbolic than a user story (e.g., \u201cwhen the SUBMIT event occurs\u201d instead of \u201cwhen the user presses the submit button\u201d), and that\u2019s for a reason. By representing states, events, and transitions symbolically, we can visualize what this state machine looks like:\n\nEvery state is represented by a box, and every event is connected to a transition arrow that connects two states. This makes it intuitive to follow the flow and understand what the next state should be given the current state and an event.\nFrom Visuals to Code\nDrawing a state machine doesn\u2019t require any special software; in fact, using paper and pencil (in case anything changes!) does the job quite nicely. However, one common problem is handoff: it doesn\u2019t matter how detailed a user story or how well-designed a visualization is, it eventually has to be coded in order for it to become part of a real application.\nWith the state machine model described above, the same visual description can be mapped directly to code. Traditionally, and as the title suggests, this is done using switch/case statements:\nfunction loginMachine(state, event) {\n switch (state) {\n case 'start':\n if (event === 'SUBMIT') {\n return 'loading';\n }\n break;\n case 'loading':\n if (event === 'RESOLVE') {\n return 'success';\n } else if (event === 'REJECT') {\n return 'error';\n }\n break;\n case 'success':\n // Accept no further events\n break;\n case 'error':\n if (event === 'SUBMIT') {\n return 'loading';\n }\n break;\n default:\n // This should never occur\n return undefined;\n }\n}\n\nconsole.log(loginMachine('start', 'SUBMIT'));\n// => 'loading'\nThis is fine (I suppose) but personally, I find it much easier to use objects:\nconst loginMachine = {\n initial: \"start\",\n states: {\n start: {\n on: { SUBMIT: 'loading' }\n },\n loading: {\n on: {\n REJECT: 'error',\n RESOLVE: 'success'\n }\n },\n error: {\n on: {\n SUBMIT: 'loading'\n }\n },\n success: {}\n }\n};\n\nfunction transition(state, event) {\n return machine\n .states[state] // Look up the state\n .on[event] // Look up the next state based on the event\n || state; // If not found, return the current state\n}\n\nconsole.log(transition('start', 'SUBMIT'));\nAs you might have noticed, the loginMachine is a plain JS object, and can be written in JSON. This is important because it allows the machine to be visualized by a 3rd-party tool, as demonstrated here:\n\nA Common Language Between Designers and Developers\nAlthough finite state machines are a fundamental part of computer science, they have an amazing potential to bridge the application specification gap between designers and developers, as well as project managers, stakeholders, and more. By designing a state machine visually and with code, designers and developers alike can:\n\nidentify all possible states, and potentially missing states\ndescribe exactly what should happen when an event occurs on a given state, and prevent that event from having unintended side-effects in other states (ever click a submit button more than once?)\neliminate impossible states and identify states that are \u201cunreachable\u201d (have no entry transition) or \u201csunken\u201d (have no exit transition)\nadd features with full confidence of knowing what other states it might affect\nsimplify redundant states or complex user flows\ncreate test paths for almost every possible user flow, and easily identify edge cases\ncollaborate better by understanding the entire application model equally.\n\nNot a New Idea\nI\u2019m not the first to suggest that state machines can help bridge the gap between design and development.\n\nVince MingPu Shao wrote an article about designing UI states and communicating with developers effectively with finite state machines\nUser flow diagrams, which visually describe the paths that a user can take through an app to achieve certain goals, are essentially state machines. Numerous tools, from Sketch plugins to standalone apps, exist for creating them.\nIn 1999, Ian Horrocks wrote a book titled \u201cConstructing the User Interface with Statecharts\u201d, which takes state machines to the next level and describes the inherent difficulties (and solutions) with creating complex UIs. The ideas in the book are still relevant today.\nMore than a decade earlier, David Harel published \u201cStatecharts: A Visual Formalism for Complex Systems\u201d, in which the statechart - an extended hierarchical state machine model - is born.\n\nState machines and statecharts have been used for complex systems and user interfaces, both physical and digital, for decades, and are especially prevalent in other industries, such as game development and embedded electronic systems. Even NASA uses statecharts for the Curiosity Rover and more, citing many benefits:\n\nVisualized modeling\nPrecise diagrams\nAutomatic code generation\nComprehensive test coverage\nAccommodation of late-breaking requirements changes\n\nMoving Forward\nIt\u2019s time that we improve how we communicate between designers and developers, much less improve the way we develop UIs to deliver the best, bug-free, optimal user experience. There is so much more to state machines and statecharts than just being a different way of designing and coding. For more resources:\n\nThe World of Statecharts is a comprehensive guide by Erik Mogensen in using statecharts in your applications\nThe Statechart Community on Spectrum is always full of interesting ideas and questions related to state machines, statecharts, and software modeling\nI gave a talk at React Rally over a year ago about how state machines (finite automata) can improve the way we develop applications. The latest one is from Reactive Conf, where I demonstrate how statecharts can be used to automatically generate test cases.\nI have also been working on XState, which is a library for \u201cstate machines and statecharts for the modern web\u201d. You can create and visualize statecharts in JavaScript, and use them in any framework (and soon enough, multiple different languages).\n\nI\u2019m excited about the future of developing web and mobile applications with statecharts, especially with regard to faster design/development cycles, auto-generated testing, better error prevention, comprehensive analytics, and even the use of model-based reinforcement learning and artificial intelligence to greatly improve the user experience.", "year": "2018", "author": "David Khourshid", "author_slug": "davidkhourshid", "published": "2018-12-12T00:00:00+00:00", "url": "https://24ways.org/2018/state-machines-in-user-interfaces/", "topic": "code"} {"rowid": 242, "title": "Creating My First Chrome Extension", "contents": "Writing a Chrome Extension isn\u2019t as scary at it seems!\nNot too long ago, I used a Chrome extension called 20 Cubed. I\u2019m far-sighted, and being a software engineer makes it difficult to maintain distance vision. So I used 20 Cubed to remind myself to look away from my screen and rest my eyes. I loved its simple interface and design. I loved it so much, I often forgot to turn it off in the middle of presentations, where it would take over my entire screen. Oops.\nUnfortunately, the developer stopped updating the extension and removed it from Chrome\u2019s extension library. I was so sad. None of the other eye rest extensions out there matched my design aesthetic, so I decided to create my own! Want to do the same?\nFortunately, Google has some respectable documentation on how to create an extension. And remember, Chrome extensions are just HTML, CSS, and JavaScript. You can add libraries and frameworks, or you can just code the \u201cold-fashioned\u201d way. Sky\u2019s the limit!\nSetup\nBut first, some things you\u2019ll need to know about before getting started:\n\nCallbacks\nTimeouts\nChrome Dev Tools\n\nDeveloping with Chrome extension methods requires a lot of callbacks. If you\u2019ve never experienced the joy of callback hell, creating a Chrome extension will introduce you to this concept. However, things can get confusing pretty quickly. I\u2019d highly recommend brushing up on that subject before getting started.\nHyperbole and a Half\nTimeouts and Intervals are another thing you might want to brush up on. While creating this extension, I didn\u2019t consider the fact that I\u2019d be juggling three timers. And I probably would\u2019ve saved time organizing those and reading up on the Chrome extension Alarms documentation beforehand. But more on that in a bit.\nOn the note of organization, abstraction is important! You might have any combination of the following:\n\nThe Chrome extension options page\nThe popup from the Chrome Menu\nThe windows or tabs you create\nThe background scripts\n\nAnd that can get unwieldy. You might also edit the existing tabs or windows in the browser, which you\u2019ll probably want as a separate script too. Note that this tutorial only covers creating your own customized window rather than editing existing windows or tabs.\nAlright, now that you know all that up front, let\u2019s get going!\nDocumentation\nTL;DR READ THE DOCS.\nA few things to get started:\n\nRead Google\u2019s primer on browser extensions\nHave a look at their Getting started tutorial\nCheck out their overview on Chrome Extensions\n\nThis overview discusses the Chrome extension files, architecture, APIs, and communication between pages. Funnily enough, I only discovered the Overview page after creating my extension.\nThe manifest.json file gives the browser information about the extension, including general information, where to find your extension files and icons, and API permissions required. Here\u2019s what my manifest.json looked like, for example:\nhttps://github.com/jennz0r/eye-rest/blob/master/manifest.json\nBecause I\u2019m a visual learner, I found the images that describe the extension\u2019s architecture most helpful.\n\nTo clarify this diagram, the background.js file is the extension\u2019s event handler. It\u2019s constantly listening for browser events, which you\u2019ll feed to it using the Chrome Extension API. Google says that an effective background script is only loaded when it is needed and unloaded when it goes idle.\nThe Popup is the little window that appears when you click on an extension\u2019s icon in the Chrome Menu. It consists of markup and scripts, and you can tell the browser where to find it in the manifest.json under page_action: { \"default_popup\": FILE_NAME_HERE }.\nThe Options page is exactly as it says. This displays customizable options only visible to the user when they either right-click on the Chrome menu and choose \u201cOptions\u201d under an extension. This also consists of markup and scripts, and you can tell the browser where to find it in the manifest.json under options_page: FILE_NAME_HERE.\nContent scripts are any scripts that will interact with any web windows or tabs that the user has open. These scripts will also interact with any tabs or windows opened by your extension.\nDebugging\nA quick note: don\u2019t forget the debugging tutorial!\nJust like any other Chrome window, every piece of an extension has an inspector and dev tools. If (read: when) you run into errors (as I did), you\u2019re likely to have several inspector windows open \u2013 one for the background script, one for the popup, one for the options, and one for the window or tab the extension is interacting with.\nFor example, I kept seeing the error \u201cThis request exceeds the MAX_WRITE_OPERATIONS_PER_HOUR quota.\u201d Well, it turns out there are limitations on how often you can sync stored information.\nAnother error I kept seeing was \u201cAlarm delay is less than minimum of 1 minutes. In released .crx, alarm \u201cALARM_NAME_HERE\u201d will fire in approximately 1 minutes\u201d. Well, it turns out there are minimum interval times for alarms.\nChrome Extension creation definitely benefits from debugging skills. Especially with callbacks and listeners, good old fashioned console.log can really help!\nMe adding a ton of `console.log`s while trying to debug my alarms.\nEye Rest Functionality\nOk, so what is the extension I created? Again, it\u2019s a way to rest your eyes every twenty minutes for twenty seconds. So, the basic functionality should look like the following:\n\nIf the extension is running AND\nIf the user has not clicked Pause in the Popup HTML AND\nIf the counter in the Popup HTML is down to 00:00 THEN\n\nOpen a new window with Timer HTML AND\nStart a 20 sec countdown in Timer HTML AND\nReset the Popup HTML counter to 20:00\n\nIf the Timer HTML is down to 0 sec THEN\n\nClose that window. Rinse. Repeat.\n\n\nSounds simple enough, but wow, these timers became convoluted! Of all the Chrome extensions I decided to create, I decided to make one that\u2019s heavily dependent on time, intervals, and having those in sync with each other. In other words, I made this unnecessarily complicated and didn\u2019t realize until I started coding.\nFor visual reference of my confusion, check out the GitHub repository for Eye Rest. (And yes, it\u2019s a pun.)\nAPI\nNow let\u2019s discuss the APIs that I used to build this extension.\nAlarms\nWhat even are alarms? I didn\u2019t know either.\nAlarms are basically Chrome\u2019s setTimeout and setInterval. They exist because, as Google says\u2026\n\nDOM-based timers, such as window.setTimeout() or window.setInterval(), are not honored in non-persistent background scripts if they trigger when the event page is dormant.\n\nFor more information, check out this background migration doc.\nOne interesting note about alarms in Chrome extensions is that they are persistent. Garbage collection with Chrome extension alarms seems unreliable at best. I didn\u2019t have much luck using the clearAll method to remove alarms I created on previous extension loads or installs. A workaround (read: hack) is to specify a unique alarm name every time your extension is loaded and clearing any other alarms without that unique name.\nBackground Scripts\nFor Eye Rest, I have two background scripts. One is my actual initializer and event listener, and the other is a helpers file.\nI wanted to share a couple of functions between my Background and Popup scripts. Specifically, the clearAndCreateAlarm function. I wanted my background script to clear any existing alarms, create a new alarm, and add remaining time until the next alarm to local storage immediately upon extension load. To make the function available to the Background script, I added helpers.js as the first item under background > scripts in my manifest.json.\nI also wanted my Popup script to do the same things when the user has unpaused the extension\u2019s functionality. To make the function available to the Popup script, I just include the helpers script in the Popup HTML file.\nOther APIs\nWindows\nI use the Windows API to create the Timer window when the time of my alarm is up. The window creation is initiated by my Background script.\nOne day, while coding late into the evening, I found it very confusing that the window.create method included url as an option. I assumed it was meant to be an external web address. A friend pondered that there must be an option to specify the window\u2019s HTML. Until then, it hadn\u2019t dawned on me that the url could be relative. Duh. I was tired!\nI pass the timer.html as the url option, as well as type, size, position, and other visual options.\nStorage\nMaybe you want to pass information back and forth between the Background script and your Popup script? You can do that using Chrome or local storage. One benefit of using local storage over Chrome\u2019s storage is avoiding quotas and write operation maximums.\nI wanted to pass the time at which the latest alarm was set, the time to the next alarm, and whether or not the timer is paused between the Background and Popup scripts. Because the countdown should change every second, it\u2019s quite complicated and requires lots of writes. That\u2019s why I went with the user\u2019s local storage. You can see me getting and setting those variables in my Background, Helper, and Popup scripts. Just search for date, nextAlarmTime, and isPaused.\nDeclarative Content\nThe Declarative Content API allows you to show your extension\u2019s page action based on several type of matches, without needing to take a host permission or inject a content script. So you\u2019ll need this to get your extension to work in the browser!\nYou can see me set this in my Background script. Because I want my extension\u2019s popup to appear on every page one is browsing, I leave the page matchers empty.\nThere are many more APIs for Chrome apps and extensions, so make sure to surf around and see what features are available!\nThe Extension\nHere\u2019s what my original Popup looked like before I added styles.\nAnd here\u2019s what it looks like with new styles. I guess I\u2019m going for a Nickelodeon feel.\nAnd here\u2019s the Timer window and Popup together! \nPublishing\nPublishing is a cinch. You just zip up your files, create a new or use an existing Google Developer account, upload the files, add some details, and pay a one time $5 fee. That\u2019s all! Then your extension will be available on the Chrome extension store! Neato :D\nMy extension is now available for you to install.\nConclusion\nI thought creating a time based Chrome Extension would be quick and easy. I was wrong. It was more complicated than I thought! But it\u2019s definitely achievable with some time, persistence, and good ole Google searches.\nEventually, I\u2019d like to add more interactive elements to Eye Rest. For example, hitting the YouTube API to grab a silly or cute video as a reward for looking away during the 20 sec countdown and not closing the timer window. This harkens back to one of my first web projects, Toothtimer, from 2012. Or maybe a way to change the background colors of the Timer and Popup!\nEither way, with Eye Rest\u2019s framework built out, I\u2019m feeling fearless about future feature adds! Building this Chrome extension took some broken nails, achy shoulders, and tired eyes, but now Eye Rest can tell me to give my eyes a break every 20 minutes.", "year": "2018", "author": "Jennifer Wong", "author_slug": "jenniferwong", "published": "2018-12-05T00:00:00+00:00", "url": "https://24ways.org/2018/my-first-chrome-extension/", "topic": "code"} {"rowid": 254, "title": "What I Learned in Six Years at GDS", "contents": "When I joined the Government Digital Service in April 2012, GOV.UK was just going into public beta. GDS was a completely new organisation, part of the Cabinet Office, with a mission to stop wasting government money on over-complicated and underperforming big IT projects and instead deliver simple, useful services for the public.\nLots of people who were experts in their fields were drawn in by this inspiring mission, and I learned loads from working with some true leaders. Here are three of the main things I learned.\n1. What is the user need?\n\u2028The main discipline I learned from my time at GDS was to always ask \u2018what is the user need?\u2019 It\u2019s very easy to build something that seems like a good idea, but until you\u2019ve identified what problem you are solving for the user, you can\u2019t be sure that you are building something that is going to help solve an actual problem.\nA really good example of this is GOV.UK Notify. This service was originally conceived of as a status tracker; a \u201cwhere\u2019s my stuff\u201d for government services. For example, if you apply for a passport online, it can take up to six weeks to arrive. After a few weeks, you might feel anxious and phone the Home Office to ask what\u2019s happening. The idea of the status tracker was to allow you to get this information online, saving your time and saving government money on call centres.\nThe project started, as all GDS projects do, with a discovery. The main purpose of a discovery is to identify the users\u2019 needs. At the end of this discovery, the team realised that a status tracker wasn\u2019t the way to address the problem. As they wrote in this blog post: \n\nStatus tracking tools are often just \u2018channel shift\u2019 for anxiety. They solve the symptom and not the problem. They do make it more convenient for people to reduce their anxiety, but they still require them to get anxious enough to request an update in the first place.\n\nWhat would actually address the user need would be to give you the information before you get anxious about where your passport is. For example, when your application is received, email you to let you know when to expect it, and perhaps text you at various points in the process to let you know how it\u2019s going. So instead of a status tracker, the team built GOV.UK Notify, to make it easy for government services to incorporate text, email and even letter notifications into their processes.\nMaking sure you know your user\nAt GDS user needs were taken very seriously. We had a user research lab on site and everyone was required to spend two hours observing user research every six weeks. Ideally you\u2019d observe users working with things you\u2019d built, but even if they weren\u2019t, it was an incredibly valuable experience, and something you should seek out if you are able to.\nEven if we think we understand our users very well, it is very enlightening to see how users actually use your stuff. Partly because in technology we tend to be power users and the average user doesn\u2019t use technology the same way we do. But even if you are building things for other developers, someone who is unfamiliar with it will interact with it in a way that may be very different to what you have envisaged.\nUser needs is not just about building things\nAsking the question \u201cwhat is the user need?\u201d really helps focus on why you are doing what you are doing. It keeps things on track, and helps the team think about what the actual desired end goal is (and should be). \nThinking about user needs has helped me with lots of things, not just building services. For example, you are raising a pull request. What\u2019s the user need? The reviewer needs to be able to easily understand what the change you are proposing is, why you are proposing that change and any areas you need particular help on with the review. \nOr you are writing an email to a colleague. What\u2019s the user need? What are you hoping the reader will learn, understand or do as a result of your email?\n2. Make things open: it makes things better\nThe second important thing I learned at GDS was \u2018make things open: it makes things better\u2019. This works on many levels: being open about your strategy, blogging about what you are doing and what you\u2019ve learned (including mistakes), and \u2013 the part that I got most involved in \u2013 coding in the open.\nTalking about your work helps clarify it\nOne thing we did really well at GDS was blogging \u2013 a lot \u2013 about what we were working on. Blogging about what you are working on is is really valuable for the writer because it forces you to think logically about what you are doing in order to tell a good story. If you are blogging about upcoming work, it makes you think clearly about why you\u2019re doing it; and it also means that people can comment on the blog post. Often people had really useful suggestions or clarifying questions.\nIt\u2019s also really valuable to blog about what you\u2019ve learned, especially if you\u2019ve made a mistake. It makes sure you\u2019ve learned the lesson and helps others avoid making the same mistakes. As well as blogging about lessons learned, GOV.UK also publishes incident reports when there is an outage or service degradation. Being open about things like this really engenders an atmosphere of trust and safe learning; which helps make things better.\nCoding in the open has a lot of benefits\nIn my last year at GDS I was the Open Source Lead, and one of the things I focused on was the requirement that all new government source code should be open. From the start, GDS coded in the open (the GitHub organisation still has the non-intuitive name alphagov, because it was created by the team doing the original Alpha of GOV.UK, before GDS was even formed).\nWhen I first joined GDS I was a little nervous about the fact that anyone could see my code. I worried about people seeing my mistakes, or receiving critical code reviews. (Setting people\u2019s mind at rest about these things is why it\u2019s crucial to have good standards around communication and positive behaviour - even a critical code review should be considerately given). \nBut I quickly realised there were huge advantages to coding in the open. In the same way as blogging your decisions makes you think carefully about whether they are good ones and what evidence you have, the fact that anyone in the world could see your code (even if, in practice, they probably won\u2019t be looking) makes everyone raise their game slightly. The very fact that you know it\u2019s open, makes you make it a bit better.\nIt helps with lots of other things as well, for example it makes it easier to collaborate with people and share your work. And now that I\u2019ve left GDS, it\u2019s so useful to be able to look back at code I worked on to remember how things worked.\nShare what you learn\nIt\u2019s sometimes hard to know where to start with being open about things, but it gets easier and becomes more natural as you practice. It helps you clarify your thoughts and follow through on what you\u2019ve decided to do. Working at GDS when this was a very important principle really helped me learn how to do this well.\n3. Do the hard work to make it simple (tech edition)\n\u2018Start with user needs\u2019 and \u2018Make things open: it makes things better\u2019 are two of the excellent government design principles. They are all good, but the third thing that I want to talk about is number 4: \u2018Do the hard work to make it simple\u2019, and specifically, how this manifests itself in the way we build technology.\nAt GDS, we worked very hard to do the hard work to make the code, systems and technology we built simple for those who came after us. For example, writing good commit messages is taken very seriously. There is commit message guidance, and it was not unusual for a pull request review to ask for a commit message to be rewritten to make a commit message clearer.\nWe worked very hard on making pull requests good, keeping the reviewer in mind and making it clear to the user how best to review it.\nReviewing others\u2019 pull requests is the highest priority so that no-one is blocked, and teams have screens showing the status of open pull requests (using fourth wall) and we even had a \u2018pull request seal\u2019, a bot that publishes pull requests to Slack and gets angry if they are uncommented on for more than two days.\nMaking it easier for developers to support the site\nAnother example of doing the hard work to make it simple was the opsmanual. I spent two years on the web operations team on GOV.UK, and one of the things I loved about that team was the huge efforts everyone went to to be open and inclusive to developers.\nThe team had some people who were really expert in web ops, but they were all incredibly helpful when bringing me on board as a developer with no previous experience of web ops, and also patiently explaining things whenever other devs in similar positions came with questions. \nThe main artefact of this was the opsmanual, which contained write-ups of how to do lots of things. One of the best things was that every alert that might lead to someone being woken up in the middle of the night had a link to documentation on the opsmanual which detailed what the alert meant and some suggested actions that could be taken to address it.\nThis was important because most of the devs on GOV.UK were on the on-call rota, so if they were woken at 3am by an alert they\u2019d never seen before, the opsmanual information might give them everything they needed to solve it, without the years of web ops training and the deep familiarity with the GOV.UK infrastructure that came with working on it every day.\nDevelopers are users too\nDoing the hard work to make it simple means that users can do what they need to do, and this applies even when the users are your developer peers. At GDS I really learned how to focus on simplicity for the user, and how much better this makes things work.\nThese three principles help us make great things\nI learned so much more in my six years at GDS. For example, the civil service has a very fair way of interviewing. I learned about the importance of good comms, working late, responsibly and the value of content design.\nAnd the real heart of what I learned, the guiding principles that help us deliver great products, is encapsulated by the three things I\u2019ve talked about here: think about the user need, make things open, and do the hard work to make it simple.", "year": "2018", "author": "Anna Shipman", "author_slug": "annashipman", "published": "2018-12-08T00:00:00+00:00", "url": "https://24ways.org/2018/what-i-learned-in-six-years-at-gds/", "topic": "business"} {"rowid": 256, "title": "Develop Your Naturalist Superpowers with Observable Notebooks and iNaturalist", "contents": "We\u2019re going to level up your knowledge of what animals you might see in an area at a particular time of year - a skill every naturalist* strives for - using technology! Using iNaturalist and Observable Notebooks we\u2019re going to prototype seasonality graphs for particular species in an area, and automatically create a guide to what animals you might see in each month.\n*(a Naturalist is someone who likes learning about nature, not someone who\u2019s a fan of being naked, that\u2019s a \u2018Naturist\u2019\u2026 different thing!)\nLooking for critters in rocky intertidal habitats\nOne of my favourite things to do is going rockpooling, or as we call it over here in California, \u2018tidepooling\u2019. Amounting to the same thing, it\u2019s going to a beach that has rocks where the tide covers then uncovers little pools of water at different times of the day. All sorts of fun creatures and life can be found in this \u2018rocky intertidal habitat\u2019\nA particularly exciting creature that lives here is the Nudibranch, a type of super colourful \u2018sea slug\u2019. There are over 3000 species of Nudibranch worldwide. (The word \u201cnudibranch\u201d comes from the Latin nudus, naked, and the Greek \u03b2\u03c1\u03b1\u03bd\u03c7\u03b9\u03b1 / brankhia, gills.)\n\u200b\n\nThey are however quite tricky to find! Even though they are often brightly coloured and interestingly shaped, some of them are very small, and in our part of the world in the Bay Area in California their appearance in our rockpools is seasonal. We see them more often in Summer months, despite the not-as-low tides as in our Winter and Spring seasons.\nMy favourite place to go tidepooling here is Pillar Point in Half Moon bay (at other times of the year more famously known for the surf competition \u2018Mavericks\u2019). The rockpools there are rich in species diversity, of varied types and water-coverage habitat zones as well as being relatively accessible.\n\u200b\n\nI was rockpooling at Pillar Point recently with my parents and we talked to a lady who remarked that she hadn\u2019t seen any Nudibranchs on her visit this time. I realised that having an idea of what species to find where, and at what time of year is one of the many superpower goals of every budding Naturalist. \nUsing technology and the croudsourced species observations of the iNaturalist community we can shortcut our way to this superpower!\nFinding nearby animals with iNaturalist\nWe\u2019re going to be getting our information about what animals you can see in Pillar Point using iNaturalist. iNaturalist is a really fun platform that helps connect people to nature and report their findings of life in the outdoors. It is also a community of nature-loving people who help each other identify and confirm those observations. iNaturalist is a project run as a joint initiative by the California Academy of Sciences and the National Geographic Society.\nI\u2019ve been using iNaturalist for over two years to record and identify plants and animals that I\u2019ve found in the outdoors. I use their iPhone app to upload my pictures, which then uses machine learning algorithms to make an initial guess at what it is I\u2019ve seen. The community is really active, and I often find someone else has verified or updated my species guess pretty soon after posting. \nThis process is great because once an observation has been identified by at least two people it becomes \u2018verified\u2019 and is considered research grade. Research grade observations get exported and used by scientists, as well as being indexed by the Global Biodiversity Information Facility, GBIF.\n\u200b\n\niNaturalist has a great API and API explorer, which makes interacting and prototyping using iNaturalist data really fun. For example, if you go to the API explorer and expand the Observations : Search and fetch section and then the GET /observations API, you get a selection of input boxes that allow you to play with options that you can then pass to the API when you click the \u2018Try it out\u2019 button.\n\u200b\n\nYou\u2019ll then get a URL that looks a bit like\nhttps://api.inaturalist.org/v1/observations?captive=false &geo=true&verifiable=true&taxon_id=47113&lat=37.495461&lng=-122.499584 &radius=5&order=desc&order_by=created_at \nwhich you can call and interrrogate using a programming language of your choice.\nIf you would like to see an all-JavaScript application that uses the iNaturalist API, take a look at OwlsNearMe.com which Simon and I built one weekend earlier this year. It gets your location and shows you all iNaturalist observations of owls near you and lists which species you are likely to see (not adjusted for season).\nRapid development using Observable Notebooks\nWe\u2019re going to be using Observable Notebooks to prototype our examples, pulling data down from iNaturalist. I really like using visual notebooks like Observable, they are great for learning and building things quickly. You may be familiar with Jupyter notebooks for Python which is similar but takes a bit of setup to get going - I often use these for prototyping too. Observable is amazing for querying and visualising data with JavaScript and since it is a hosted product it doesn\u2019t require any setup at all.\nYou can follow along and play with this example on my Observable notebook. If you create an account there you can fork my notebook and create your own version of this example. \nEach \u2018notebook\u2019 consists of a page with a column of \u2018cells\u2019, similar to what you get in a spreadsheet. A cell can contain Markdown text or JavaScript code and the output of evaluating the cell appears above the code that generated it. There are lots of tutorials out there on Observable Notebooks, I like this code introduction one from Observable (and D3) creator Mike Bostock.\nDeveloping your Naturalist superpowers\nIf you have an idea of what plants and critters you might see in a place at the time you visit, you can hone in on what you want to study and train your Naturalist eye to better identify the life around you.\nFor our example, we care about wildlife we can see at Pillar Point, so we need a way of letting the iNaturalist API know which area we are interested in.\nWe could use a latitide, longitude and radius for this, but a rectangular bounding box is a better shape for the reef. We can use this tool to draw the area we want to search within: boundingbox.klokantech.com\n\u200b\n\nThe tool lets you export the bounding box in several forms using the dropdown at the bottom left under the map givese We are going to use the \u2018DublinCore\u2019 format as it\u2019s closest to the format needed by the iNaturalist API.\n westlimit=-122.50542; southlimit=37.492805; eastlimit=-122.492738; northlimit=37.499811\nA quick map primer:\nThe higher the latitude the more north it is\nThe lower the latitude the more south it is\nLatitude 0 = the equator\n\nThe higher the longitude the more east it is of Greenwich\nThe lower the longitude the more west it is of Greenwich\nLongitude 0 = Greenwich\nIn the iNaturalst API we want to use the parameters nelat, nelng, swlat, swlng to create a query that looks inside a bounding box of Pillar Point near Half Moon Bay in California:\nnelat = highest latitude = north limit = 37.499811\nnelng = highest longitude = east limit = -122.492738\nswlat = smallest latitude = south limit = 37.492805\nswlng = smallest longitude = west limit = 122.50542\nAs API parameters these look like this:\n?nelat=37.499811&nelng=-122.492738&swlat=37.492805&swlng=122.50542\nThese parameters in this format can be used for most of the iNaturalist API methods.\nNudibranch seasonality in Pillar Point\nWe can use the iNaturalist observation_histogram API to get a count of Nudibranch observations per week-of-year across all time and within our Pillar Point bounding box.\nIn addition to the geographic parameters that we just worked out, we are also sending the taxon_id of 47113, which is iNaturalists internal number associated with the Nudibranch taxon. By using this we can get all species which are under the parent \u2018Order Nudibranchia\u2019. \nAnother useful piece of naturalist knowledge is understanding the biological classification scheme of Taxanomic Rank - roughly, when a species has a Latin name of two words eg \u2018Glaucus Atlanticus\u2019 the first Latin word is the \u2018Genus\u2019 like a family name \u2018Glaucus\u2019, and the second word identifies that particular species, like a given name \u2018Atlanticus\u2019. \nThe two Latin words together indicate a specific species, the term we use colloquially to refer to a type of animal often differs wildly region to region, and sometimes the same common name in two countries can refer to two different species. The common names for the Glaucus Atlanticus (which incidentally is my favourite sea slug) include: sea swallow, blue angel, blue glaucus, blue dragon, blue sea slug and blue ocean slug! Because this gets super confusing, Scientists like using this Latin name format instead.\nThe following piece of code asks the iNaturalist Histogram API to return per-week counts for verified observations of Nudibranchs within our Pillar Point bounding box:\npillar_point_counts_per_week = fetch(\n \"https://api.inaturalist.org/v1/observations/histogram?taxon_id=47113&nelat=37.499811&nelng=-122.492738&swlat=37.492805&swlng=-122.50542&date_field=observed&interval=week_of_year&verifiable=true\"\n ).then(response => {\n return response.json();\n})\nOur next step is to take this data and draw a graph! We\u2019ll be using Vega-Lite for this, which is a fab JavaScript graphing libary that is also easy and fun to use with Observable Notebooks. \n(Here is a great tutorial on exploring data and drawing graphs with Observable and Vega-Lite)\nThe iNaturalist API returns data that looks like this:\n{\n \"total_results\": 53,\n \"page\": 1,\n \"per_page\": 53,\n \"results\": {\n \"week_of_year\": {\n \"1\": 136,\n \"2\": 20,\n \"3\": 150,\n \"4\": 65,\n \"5\": 186,\n \"6\": 74,\n \"7\": 47,\n \"8\": 87,\n \"9\": 64,\n \"10\": 56,\nBut for our Vega-Lite graph we need data that looks like this:\n[{\n \"week\": \"01\",\n \"value\": 136\n}, {\n \"week\": \"02\",\n \"value\": 20\n}, ...]\nWe can convert what we get back from the API to the second format using a loop that iterates over the object keys:\nobjects_to_plot = {\n let objects = [];\n Object.keys(pillar_point_counts_per_week.results.week_of_year).map(function(week_index) {\n objects.push({\n week: `Wk ${week_index.toString()}`,\n observations: pillar_point_counts_per_week.results.week_of_year[week_index]\n });\n })\n return objects;\n}\nWe can then plug this into Vega-Lite to draw us a graph:\nvegalite({\n data: {values: objects_to_plot},\n mark: \"bar\",\n encoding: {\n x: {field: \"week\", type: \"nominal\", sort: null},\n y: {field: \"observations\", type: \"quantitative\"}\n },\n width: width * 0.9\n})\n\nIt\u2019s worth noting that we have a lot of observations of Nudibranchs particularly at Pillar Point due in no small part to the intertidal monitoring research that Alison Young and Rebecca Johnson facilitate for the California Achademy of Sciences. \nSo, what if we want to look for the seasonality of observations of a particular species of adorable sea slug? We want our interface to have a select box with a list of all the species you might find at any time of year. We can do this using the species_counts API to create us an object with the iNaturalist species ID and common & Latin names.\npillar_point_nudibranches = {\n let api_results = await fetch(\n \"https://api.inaturalist.org/v1/observations/species_counts?taxon_id=47113&nelat=37.499811&nelng=-122.492738&swlat=37.492805&swlng=-122.50542&date_field=observed&verifiable=true\"\n ).then(r => r.json())\n\n let species_list = api_results.results.map(i => ({\n value: i.taxon.id,\n label: `${i.taxon.preferred_common_name} (${i.taxon.name})`\n }));\n\n return species_list\n}\nWe can create an interactive select box by importing code from Jeremy Ashkanas\u2019 Observable Notebook: add import {select} from \"@jashkenas/inputs\" to a cell anywhere in our notebook. Observable is magic: like a spreadsheet, the order of the cells doesn\u2019t matter - if one cell is referenced by any other cell then when that cell updates all the other cells refresh themselves. You can also import and reference one notebook from another!\nviewof select_species = select({\n title: \"Which Nudibranch do you want to see seasonality for?\",\n options: [{value: \"\", label: \"All the Nudibranchs!\"}, ...pillar_point_nudibranches],\n value: \"\"\n})\nThen we go back to our old favourite, the histogram API just like before, only this time we are calling it with the value created by our select box ${select_species} as taxon_id instead of the number 47113.\npillar_point_counts_per_month_per_species = fetch(\n `https://api.inaturalist.org/v1/observations/histogram?taxon_id=${select_species}&nelat=37.499811&nelng=-122.492738&swlat=37.492805&swlng=-122.50542&date_field=observed&interval=month_of_year&verifiable=true`\n).then(r => r.json())\nNow for the fun graph bit! As we did before, we re-format the result of the API into a format compatible with Vega-Lite:\nobjects_to_plot_species_month = {\n let objects = [];\n Object.keys(pillar_point_counts_per_month_per_species.results.month_of_year).map(function(month_index) {\n objects.push({\n month: (new Date(2018, (month_index - 1), 1)).toLocaleString(\"en\", {month: \"long\"}),\n observations: pillar_point_counts_per_month_per_species.results.month_of_year[month_index]\n });\n })\n return objects;\n}\n(Note that in the above code we are creating a date object with our specific month in, and using toLocalString() to get the longer English name for the month. Because the JavaScript Date object counts January as 0, we use month_index -1 to get the correct month)\nAnd we draw the graph as we did before, only now if you interact with the select box in Observable the graph will dynamically update!\nvegalite({\n data: {values: objects_to_plot_species_month},\n mark: \"bar\",\n encoding: {\n x: {field: \"month\", type: \"nominal\", sort:null},\n y: {field: \"observations\", type: \"quantitative\"}\n },\n width: width * 0.9\n})\nNow we can see when is the best time of year to plan to go tidepooling in Pillar Point if we want to find a specific species of Nudibranch.\n\u200b\n\nThis tool is great for planning when we to go rockpooling at Pillar Point, but what about if you are going this month and want to pre-train your eye with what to look for in order to impress your friends with your knowledge of Nudibranchs?\nWell\u2026 we can create ourselves a dynamic guide that you can with a list of the species, their photo, name and how many times they have been observed in that month of the year!\nOur select box this time looks as follows, simpler than before but assigning the month value to the variable selected_month.\nviewof selected_month = select({\n title: \"When do you want to see Nudibranchs?\",\n options: [\n { label: \"Whenever\", value: \"\" },\n { label: \"January\", value: \"1\" },\n { label: \"February\", value: \"2\" },\n { label: \"March\", value: \"3\" },\n { label: \"April\", value: \"4\" },\n { label: \"May\", value: \"5\" },\n { label: \"June\", value: \"6\" },\n { label: \"July\", value: \"7\" },\n { label: \"August\", value: \"8\" },\n { label: \"September\", value: \"9\" },\n { label: \"October\", value: \"10\" },\n { label: \"November\", value: \"11\" },\n { label: \"December\", value: \"12\" },\n ],\n value: \"\"\n })\nWe then can use the species_counts API to get all the relevant information about which species we can see in month=${selected_month}. We\u2019ll be able to reference this response object and its values later with the variable we just created, eg: all_species_data.results[0].taxon.name.\nall_species_data = fetch(\n `https://api.inaturalist.org/v1/observations/species_counts?taxon_id=47113&month=${selected_month}&nelat=37.499811&nelng=-122.492738&swlat=37.492805&swlng=-122.50542&verifiable=true`\n).then(r => r.json())\nYou can render HTML directly in a notebook cell using Observable\u2019s html tagged template literal:\n\n\n
Seen ${s.count} times
\n