rowid,title,contents,year,author,author_slug,published,url,topic
76,Giving CSS Animations and Transitions Their Place,"CSS animations and transitions may not sit squarely in the realm of the behaviour layer, but they’re stepping up into this area that used to be pure JavaScript territory. Heck, CSS might even perform better than its JavaScript equivalents in some cases. That’s pretty serious! With CSS’s new tricks blurring the lines between presentation and behaviour, it can start to feel bloated and messy in our CSS files. It’s an uncomfortable feeling.
Here are a pair of methods I’ve found to be pretty helpful in keeping the potential bloat and wire-crossing under control when CSS has its hands in both presentation and behaviour.
Same eggs, more baskets
Structuring your CSS to have separate files for layout, typography, grids, and so on is a fairly common approach these days. But which one do you put your transitions and animations in? The initial answer, as always, is “it depends”.
Small effects here and there will likely sit just fine with your other styles. When you move into more involved effects that require multiple animations and some logic support from JavaScript, it’s probably time to choose none of the above, and create a separate CSS file just for them.
Putting all your animations in one file is a huge help for code organization. Even if you opt for a name less literal than animations.css, you’ll know exactly where to go for anything CSS animation related. That saves time and effort when it comes to editing and maintenance. Keeping track of which animations are still currently used is easier when they’re all grouped together as well. And as an added bonus, you won’t have to look at all those horribly unattractive and repetitive prefixed @-keyframe rules unless you actually need to.
An animations.css file might look something like the snippet below. It defines each animation’s keyframes and defines a class for each variation of that animation you’ll be using. Depending on the situation, you may also want to include transitions here in a similar way. (I’ve found defining transitions as their own class, or mixin, to be a huge help in past projects for me.)
// defining the animation
@keyframes catFall {
from { background-position: center 0;}
to {background-position: center 1000px;}
}
@-webkit-keyframes catFall {
from { background-position: center 0;}
to {background-position: center 1000px;}
}
@-moz-keyframes catFall {
from { background-position: center 0;}
to {background-position: center 1000px;}
}
@-ms-keyframes catFall {
from { background-position: center 0;}
to {background-position: center 1000px;}
}
…
// class that assigns the animation
.catsBackground {
height: 100%;
background: transparent url(../endlessKittens.png) 0 0 repeat-y;
animation: catFall 1s linear infinite;
-webkit-animation: catFall 1s linear infinite;
-moz-animation: catFall 1s linear infinite;
-ms-animation: catFall 1s linear infinite;
}
If we don’t need it, why load it?
Having all those CSS animations and transitions in one file gives us the added flexibility to load them only when we want to. Loading a whole lot of things that will never be used might seem like a bit of a waste.
While CSS has us impressed with its motion chops, it falls flat when it comes to the logic and fine-grained control. JavaScript, on the other hand, is pretty good at both those things. Chances are the content of your animations.css file isn’t acting alone. You’ll likely be adding and removing classes via JavaScript to manage your CSS animations at the very least. If your CSS animations are so entwined with JavaScript, why not let them hang out with the rest of the behaviour layer and only come out to play when JavaScript is supported?
Dynamically linking your animations.css file like this means it will be completely ignored if JavaScript is off or not supported. No JavaScript? No additional behaviour, not even the parts handled by CSS.
This technique comes up in progressive enhancement techniques as well, but it can help here to keep your presentation and behaviour nicely separated when more than one language is involved. The aim in both cases is to avoid loading files we won’t be using.
If you happen to be doing something a bit fancier – like 3-D transforms or critical animations that require more nuanced fallbacks – you might need something like modernizr to step in to determine support more specifically. But the general idea is the same.
Summing it all up
Using a couple of simple techniques like these, we get to pick where to best draw the line between behaviour and presentation based on the situation at hand, not just on what language we’re using. The power of when to separate and how to reassemble the individual pieces can be even greater if you use preprocessors as part of your process. We’ve got a lot of options! The important part is to make forward-thinking choices to save your future self, and even your current self, unnecessary headaches.",2012,Val Head,valhead,2012-12-08T00:00:00+00:00,https://24ways.org/2012/giving-css-animations-and-transitions-their-place/,code
175,Front-End Code Reusability with CSS and JavaScript,"Most web standards-based developers are more than familiar with creating their sites with semantic HTML with lots and lots of CSS. With each new page in a design, the CSS tends to grow and grow and more elements and styles are added. But CSS can be used to better effect.
The idea of object-oriented CSS isn’t new. Nicole Sullivan has written a presentation on the subject and outlines two main concepts: separate structure and visual design; and separate container and content. Jeff Croft talks about Applying OOP Concepts to CSS:
I can make a class of .box that defines some basic layout structure, and another class of .rounded that provides rounded corners, and classes of .wide and .narrow that define some widths, and then easily create boxes of varying widths and styles by assigning multiple classes to an element, without having to duplicate code in my CSS.
This concept helps reduce CSS file size, allows for great flexibility, rapid building of similar content areas and means greater consistency throughout the entire design. You can also take this concept one step further and apply it to site behaviour with JavaScript.
Build a versatile slideshow
I will show you how to build multiple slideshows using jQuery, allowing varying levels of functionality which you may find on one site design. The code will be flexible enough to allow you to add previous/next links, image pagination and the ability to change the animation type. More importantly, it will allow you to apply any combination of these features.
Image galleries are simply a list of images, so the obvious choice of marking the content up is to use a
. Many designs, however, do not cater to non-JavaScript versions of the website, and thus don’t take in to account large multiple images. You could also simply hide all the other images in the list, apart from the first image. This method can waste bandwidth because the other images might be downloaded when they are never going to be seen.
Taking this second concept — only showing one image — the only code you need to start your slideshow is an tag. The other images can be loaded dynamically via either a per-page JavaScript array or via AJAX.
The slideshow concept is built upon the very versatile Cycle jQuery Plugin and is structured in to another reusable jQuery plugin. Below is the HTML and JavaScript snippet needed to run every different type of slideshow I have mentioned above.
Slideshow plugin
If you’re not familiar with jQuery or how to write and author your own plugin there are plenty of articles to help you out.
jQuery has a chainable interface and this is something your plugin must implement. This is easy to achieve, so your plugin simply returns the collection it is using:
return this.each(
function () {}
};
Local Variables
To keep the JavaScript clean and avoid any conflicts, you must set up any variables which are local to the plugin and should be used on each collection item. Defining all your variables at the top under one statement makes adding more and finding which variables are used easier. For other tips, conventions and improvements check out JSLint, the “JavaScript Code Quality Tool”.
var $$, $div, $images, $arrows, $pager,
id, selector, path, o, options,
height, width,
list = [], li = 0,
parts = [], pi = 0,
arrows = ['Previous', 'Next'];
Cache jQuery Objects
It is good practice to cache any calls made to jQuery. This reduces wasted DOM calls, can improve the speed of your JavaScript code and makes code more reusable.
The following code snippet caches the current selected DOM element as a jQuery object using the variable name $$. Secondly, the plugin makes its settings available to the Metadata plugin‡ which is best practice within jQuery plugins.
For each slideshow the plugin generates a
with a class of slideshow and a unique id. This is used to wrap the slideshow images, pagination and controls.
The base path which is used for all the images in the slideshow is calculated based on the existing image which appears on the page. For example, if the path to the image on the page was /img/flowers/1.jpg the plugin would use the path /img/flowers/ to load the other images.
$$ = $(this);
o = $.metadata ? $.extend({}, settings, $$.metadata()) : settings;
id = 'slideshow-' + (i++ + 1);
$div = $('').addClass('slideshow').attr('id', id);
selector = '#' + id + ' ';
path = $$.attr('src').replace(/[0-9]\.jpg/g, '');
options = {};
height = $$.height();
width = $$.width();
Note: the plugin uses conventions such as folder structure and numeric filenames. These conventions help with the reusable aspect of plugins and best practices.
Build the Images
The cycle plugin uses a list of images to create the slideshow. Because we chose to start with one image we must now build the list programmatically. This is a case of looping through the images which were added via the plugin options, building the appropriate HTML and appending the resulting
to the DOM.
$.each(o.images, function () {
list[li++] = '
';
list[li++] = '';
list[li++] = '
';
});
$images = $('
').addClass('cycle-images');
$images.append(list.join('')).appendTo($div);
Although jQuery provides the append method it is much faster to create one really long string and append it to the DOM at the end.
Update the Options
Here are some of the options we’re making available by simply adding classes to the . You can change the slideshow effect from the default fade to the sliding effect. By adding the class of stopped the slideshow will not auto-play and must be controlled via pagination or previous and next links.
// different effect
if ($$.is('.slide')) {
options.fx = 'scrollHorz';
}
// don't move by default
if ($$.is('.stopped')) {
options.timeout = 0;
}
If you are using the same set of images throughout a website you may wish to start on a different image on each page or section. This can be easily achieved by simply adding the appropriate starting class to the .
// based on the class name on the image
if ($$.is('[class*=start-]')) {
options.startingSlide = parseInt($$.attr('class').replace(/.*start-([0-9]+).*/g, ""$1""), 10) - 1;
}
For example:
By default, and without JavaScript, the third image in this slideshow is shown. When the JavaScript is applied to the page the slideshow must know to start from the correct place, this is why the start class is required.
You could capture the default image name and parse it to get the position, but only the default image needs to be numeric to work with this plugin (and could easily be changed in future). Therefore, this extra specifically defined option means the plugin is more tolerant.
Previous/Next Links
A common feature of slideshows is previous and next links enabling the user to manually progress the images. The Cycle plugin supports this functionality, but you must generate the markup yourself. Most people add these directly in the HTML but normally only support their behaviour when JavaScript is enabled. This goes against progressive enhancement. To keep with the best practice progress enhancement method the previous/next links should be generated with JavaScript.
The follow snippet checks whether the slideshow requires the previous/next links, via the arrows class. It restricts the Cycle plugin to the specific slideshow using the selector we created at the top of the plugin. This means multiple slideshows can run on one page without conflicting each other.
The code creates a
using the arrows array we defined at the top of the plugin. It also adds a class to the slideshow container, meaning you can style different combinations of options in your CSS.
// create the arrows
if ($$.is('.arrows') && list.length > 1) {
options.next = selector + '.next';
options.prev = selector + '.previous';
$arrows = $('
').addClass('cycle-arrows');
$.each(arrows, function (i, val) {
parts[pi++] = '
';
});
$arrows.append(parts.join('')).appendTo($div);
$div.addClass('has-cycle-arrows');
}
The arrow array could be placed inside the plugin settings to allow for localisation.
Pagination
The Cycle plugin creates its own HTML for the pagination of the slideshow. All our plugin needs to do is create the list and selector to use. This snippet creates the pagination container and appends it to our specific slideshow container. It sets the Cycle plugin pager option, restricting it to the specific slideshow using the selector we created at the top of the plugin. Like the previous/next links, a class is added to the slideshow container allowing you to style the slideshow itself differently.
// create the clickable pagination
if ($$.is('.pagination') && list.length > 1) {
options.pager = selector + '.cycle-pagination';
$pager = $('
').addClass('cycle-pagination');
$pager.appendTo($div);
$div.addClass('has-cycle-pagination');
}
Note: the Cycle plugin creates a
with anchors listed directly inside without the surrounding
. Unfortunately this is invalid markup but the code still works.
Demos
Well, that describes all the ins-and-outs of the plugin, but demos make it easier to understand! Viewing the source on the demo page shows some of the combinations you can create with a simple , a few classes and some thought-out JavaScript.
View the demos →
Decide on defaults
The slideshow plugin uses the exact same settings as the Cycle plugin, but some are explicitly set within the slideshow plugin when using the classes you have set.
When deciding on what functionality is going to be controlled via this class method, be careful to choose your defaults wisely. If all slideshows should auto-play, don’t make this an option — make the option to stop the auto-play. Similarly, if every slideshow should have previous/next functionality make this the default and expose the ability to remove them with a class such as “no-pagination”.
In the examples presented on this article I have used a class on each . You can easily change this to anything you want and simply apply the plugin based on the jQuery selector required.
Grab your images
If you are using AJAX to load in your images, you can speed up development by deciding on and keeping to a folder structure and naming convention. There are two methods: basing the image path based on the current URL; or based on the src of the image. The first allows a different slideshow on each page, but in many instances a site will have a couple of sets of images and therefore the second method is probably preferred.
Metadata ‡
A method which allows you to directly modify settings in certain plugins, which also uses the classes from your HTML already exists. This is a jQuery plugin called Metadata. This method allows for finer control over the plugin settings themselves. Some people, however, may dislike the syntax and prefer using normal classes, like above which when sprinkled with a bit more JavaScript allows you to control what you need to control.
The takeaway
Hopefully you have understood not only what goes in to a basic jQuery plugin but also learnt a new and powerful idea which you can apply to other areas of your website.
The idea can also be applied to other common interfaces such as lightboxes or mapping services such as Google Maps — for example creating markers based on a list of places, each with different pin icons based the anchor class.",2009,Trevor Morris,trevormorris,2009-12-06T00:00:00+00:00,https://24ways.org/2009/front-end-code-reusability-with-css-and-javascript/,code
11,JavaScript: Taking Off the Training Wheels,"JavaScript is the third pillar of front-end web development. Of those pillars, it is both the most powerful and the most complex, so it’s understandable that when 24 ways asked, “What one thing do you wish you had more time to learn about?”, a number of you answered “JavaScript!”
This article aims to help you feel happy writing JavaScript, and maybe even without libraries like jQuery. I can’t comprehensively explain JavaScript itself without writing a book, but I hope this serves as a springboard from which you can jump to other great resources.
Why learn JavaScript?
So what’s in it for you? Why take the next step and learn the fundamentals?
Confidence with jQuery
If nothing else, learning JavaScript will improve your jQuery code; you’ll be comfortable writing jQuery from scratch and feel happy bending others’ code to your own purposes. Writing efficient, fast and bug-free jQuery is also made much easier when you have a good appreciation of JavaScript, because you can look at what jQuery is really doing. Understanding how JavaScript works lets you write better jQuery because you know what it’s doing behind the scenes. When you need to leave the beaten track, you can do so with confidence.
In fact, you could say that jQuery’s ultimate goal is not to exist: it was invented at a time when web APIs were very inconsistent and hard to work with. That’s slowly changing as new APIs are introduced, and hopefully there will come a time when jQuery isn’t needed.
An example of one such change is the introduction of the very useful document.querySelectorAll. Like jQuery, it converts a CSS selector into a list of matching elements. Here’s a comparison of some jQuery code and the equivalent without.
$('.counter').each(function (index) {
$(this).text(index + 1);
});
var counters = document.querySelectorAll('.counter');
[].slice.call(counters).forEach(function (elem, index) {
elem.textContent = index + 1;
});
Solving problems no one else has!
When you have to go to the internet to solve a problem, you’re forever stuck reusing code other people wrote to solve a slightly different problem to your own. Learning JavaScript will allow you to solve problems in your own way, and begin to do things nobody else ever has.
Node.js
Node.js is a non-browser environment for running JavaScript, and it can do just about anything! But if that sounds daunting, don’t worry: the Node community is thriving, very friendly and willing to help.
I think Node is incredibly exciting. It enables you, with one language, to build complete websites with complex and feature-filled front- and back-ends. Projects that let users log in or need a database are within your grasp, and Node has a great ecosystem of library authors to help you build incredible things. Exciting!
Here’s an example web server written with Node. http is a module that allows you to create servers and, like jQuery’s $.ajax, make requests. It’s a small amount of code to do something complex and, while working with Node is different from writing front-end code, it’s certainly not out of your reach.
var http = require('http');
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end('Hello World');
}).listen(1337);
console.log('Server running at http://localhost:1337/');
Grunt and other website tools
Node has brought in something of a renaissance in tools that run in the command line, like Yeoman and Grunt. Both of these rely heavily on Node, and I’ll talk a little bit about Grunt here.
Grunt is a task runner, and many people use it for compiling Sass or compressing their site’s JavaScript and images. It’s pretty cool. You configure Grunt via the gruntfile.js, so JavaScript skills will come in handy, and since Grunt supports plug-ins built with JavaScript, knowing it unlocks the bucketloads of power Grunt has to offer.
Ways to improve your skills
So you know you want to learn JavaScript, but what are some good ways to learn and improve? I think the answer to that is different for different people, but here are some ideas.
Rebuild a jQuery app
Converting a jQuery project to non-jQuery code is a great way to explore how you modify elements on the page and make requests to the server for data. My advice is to focus on making it work in one modern browser initially, and then go cross-browser if you’re feeling adventurous. There are many resources for directly comparing jQuery and non-jQuery code, like Jeffrey Way’s jQuery to JavaScript article.
Find a mentor
If you think you’d work better on a one-to-one basis then finding yourself a mentor could be a brilliant way to learn. The JavaScript community is very friendly and many people will be more than happy to give you their time. I’d look out for someone who’s active and friendly on Twitter, and does the kind of work you’d like to do. Introduce yourself over Twitter or send them an email. I wouldn’t expect a full tutoring course (although that is another option!) but they’ll be very glad to answer a question and any follow-ups every now and then.
Go to a workshop
Many conferences and local meet-ups run workshops, hosted by experts in a particular field. See if there’s one in your area. Workshops are great because you can ask direct questions, and you’re in an environment where others are learning just like you are — no need to learn alone!
Set yourself challenges
This is one way I like to learn new things. I have a new thing that I’m not very good at, so I pick something that I think is just out of my reach and I try to build it. It’s learning by doing and, even if you fail, it can be enormously valuable.
Where to start?
If you’ve decided learning JavaScript is an important step for you, your next question may well be where to go from here.
I’ve collected some links to resources I know of or use, with some discussion about why you might want to check a particular site out. I hope this serves as a springboard for you to go out and learn as much as you want.
Beginner
If you’re just getting started with JavaScript, I’d recommend heading to one of these places. They cover the basics and, in some cases, a little more advanced stuff. They’re all reputable sources (although, I’ve included something I wrote — you can decide about that one!) and will not lead you astray.
jQuery’s JavaScript 101 is a great first resource for JavaScript that will give you everything you need to work with jQuery like a pro.
Codecademy’s JavaScript Track is a small but useful JavaScript course. If you like learning interactively, this could be one for you.
HTMLDog’s JavaScript Tutorials take you right through from the basics of code to a brief introduction to newer technology like Node and Angular. [Disclaimer: I wrote this stuff, so it comes with a hazard warning!]
The tuts+ jQuery to JavaScript mentioned earlier is great for seeing how jQuery code looks when converted to pure JavaScript.
Getting in-depth
For more comprehensive documentation and help I’d recommend adding these places to your list of go-tos.
MDN: the Mozilla Developer Network is the first place I go for many JavaScript questions. I mostly find myself there via a search, but it’s a great place to just go and browse.
Axel Rauschmayer’s 2ality is a stunning collection of articles that will take you deep into JavaScript. It’s certainly worth looking at.
Addy Osmani’s JavaScript Design Patterns is a comprehensive collection of patterns for writing high quality JavaScript, particularly as you (I hope) start to write bigger and more complex applications.
And finally…
I think the key to learning anything is curiosity and perseverance. If you have a question, go out and search for the answer, even if you have no idea where to start. Keep going and going and eventually you’ll get there. I bet you’ll learn a whole lot along the way. Good luck!
Many thanks to the people who gave me their time when I was working on this article: Tom Oakley, Jack Franklin, Ben Howdle and Laura Kalbag.",2013,Tom Ashworth,tomashworth,2013-12-05T00:00:00+00:00,https://24ways.org/2013/javascript-taking-off-the-training-wheels/,code
294,New Tricks for an Old Dog,"Much of my year has been spent helping new team members find their way around the expansive and complex codebase that is the TweetDeck front-end, trying to build a happy and productive group of people around a substantial codebase with many layers of legacy.
I’ve loved doing this. Everything from writing new documentation, drawing diagrams, and holding technical architecture sessions teaches you something you didn’t know or exposes an area of uncertainty that you can go work on.
In this article, I hope to share some experiences and techniques that will prove useful in your own situation and that you can impress your friends in some new and exciting ways!
How do you do, fellow kids?
To start with I’d like to introduce you to our JavaScript framework, Flight. Right now it’s used by twitter.com and TweetDeck although, as a company, Twitter is largely moving to React.
Over time, as we used Flight for more complex interfaces, we found it wasn’t scaling with us.
Composing components into trees was fiddly and often only applied for a specific parent-child pairing. It seems like an obvious feature with hindsight, but it didn’t come built-in to Flight, and it made reusing components a real challenge.
There was no standard way to manage the state of a component; they all did it slightly differently, and the technique often varied by who was writing the code. This cost us in maintainability as you just couldn’t predict how a component would be built until you opened it.
Making matters worse, Flight relied on events to move data around the application. Unfortunately, events aren’t good for giving structure to complex logic. They jump around in a way that’s hard to understand and debug, and force you to search your code for a specific string — the event name‚ to figure out what’s going on.
To find fixes for these problems, we looked around at other frameworks. We like React for it’s simple, predictable state management and reactive re-render flow, and Elm for bringing strict functional programming to everyone.
But when you have lots of existing code, rewriting or switching framework is a painful and expensive option. You have to understand how it will interact with your existing code, how you’ll test it alongside existing code, and how it will affect the size and performance of the application. This all takes time and effort!
Instead of planning a rewrite, we looked for the ideas hidden within other frameworks that we could reapply in our own situation or bring to the tools we already were using.
Boiled down, what we liked seemed quite simple:
Component nesting & composition
Easy, predictable state management
Normal functions for data manipulation
Making these ideas applicable to Flight took some time, but we’re in a much better place now. Through persistent trial-and-error, we have well documented, testable and standard techniques for creating complex component hierarchies, updating and reacting to state changes, and passing data around the app.
While the specifics of our situation and Flight aren’t really important, this experience taught me something:
Distill good tech into great ideas. You can apply great ideas anywhere.
You don’t have to use cool kids’ latest framework, hottest build tool or fashionable language to benefit from them. If you can identify a nugget of gold at the heart of it all, why not use it to improve what you have already?
Times, they are a changin’
Apart from stealing ideas from the new and shiny, how can we keep make the most of improved tooling and techniques? Times change and so should the way we write code.
Going back in time a bit, TweetDeck used some slightly outmoded tools for building and bundling. Without a transpiler like Babel we were missing out new language features, and without a more advanced build tools like Webpack, every module’s source was encased in AMD boilerplate.
In fact, we found ourselves with a mix of both AMD syntaxes:
define([""lodash""], function (_) {
// . . .
});
define(function (require) {
var _ = require(""lodash"");
// . . .
});
This just wouldn’t do. And besides, what we really wanted was CommonJS, or even ES2015 module syntax:
import _ from ""lodash"";
These days we’re using Babel, Webpack, ES2015 modules and many new language features that make development just… better. But how did we get there?
To explain, I want to introduce you to codemods and jscodeshift.
A codemod is a large-scale refactor of a whole codebase, often mechanical or repetitive. Think of renaming a module or changing an API like URL(""..."") to new URL(""..."").
jscodeshift is a toolkit for running automated codemods, where you express a code transformation using code. The automated codemod operates on each file’s syntax tree – a data-structure representation of the code — finding and modifying in place as it goes.
Here’s an example that renames all instances of the variable foo to bar:
module.exports = function (fileInfo, api) {
return api
.jscodeshift(fileInfo.source)
.findVariableDeclarators('foo')
.renameTo('bar')
.toSource();
};
It’s a seriously powerful tool, and we’ve used it to write a series of codemods that:
rename modules,
unify our use of AMD to a single syntax,
transition from one testing framework to another, and
switch from AMD to CommonJS.
These changes can be pretty huge and far-reaching. Here’s an example commit from when we switched to CommonJS:
commit 8f75de8fd4c702115c7bf58febba1afa96ae52fc
Date: Tue Jul 12 2016
Run AMD -> CommonJS codemod
418 files changed, 47550 insertions(+), 48468 deletions(-)
Yep, that’s just under 50k lines changed, tested, merged and deployed without any trouble. AMD be gone!
From this step-by-step approach, using codemods to incrementally tweak and improve, we extracted a little codemod recipe for making significant, multi-stage changes:
Find all the existing patterns
Choose the two most similar
Unify with a codemod
Repeat.
For example:
For module loading, we had 2 competing AMD patterns plus some use of CommonJS
The two AMD syntaxes were the most similar
We used a codemod to move to unify the AMD patterns
Later we returned to AMD to convert it to CommonJS
It’s worked for us, and if you’d like to know more about codemods then check out Evolving Complex Systems Incrementally by Facebook engineer, Christoph Pojer.
Welcome aboard!
As TweetDeck has gotten older and larger, the amount of things a new engineer has to learn about has exploded. The myriad of microservices that manage our data and their layers of authentication, security and business logic around them make for an overwhelming amount of information to hand to a newbie.
Inspired by Amy’s amazing Guide to the Care and Feeding of Junior Devs, we realised it was important to take time to design our onboarding that each of our new hires go through to make the most of their first few weeks.
Joining a new company, team, or both, is stressful and uncomfortable. Everything you can do to help a new hire will be valuable to them. So please, take time to design your onboarding!
And as you build up an onboarding process, you’ll create things that are useful for more than just new hires; it’ll force you to write documentation, for example, in a way that’s understandable for people who are unfamiliar with your team, product and codebase. This can lead to more outside contributions: potential contributors feel more comfortable getting set up on your product without asking for help.
This is something that’s taken for granted in open source, but somehow I think we forget about it in big companies.
After all, better documentation is just a good thing. You will forget things from time to time, and you’d be surprised how often the “beginner” docs help!
For TweetDeck, we put together system and architecture diagrams, and one-pager explanations of important concepts:
What are our dependencies?
Where are the potential points of failure?
Where does authentication live? Storage? Caching?
Who owns “X”?
Of course, learning continues long after onboarding. The landscape is constantly shifting; old services are deprecated, new APIs appear and what once true can suddenly be very wrong. Keeping up with this is a serious challenge, and more than any one person can track.
To address this, we’ve thought hard about our knowledge sharing practices across the whole team. For example, we completely changed the way we do code review.
In my opinion, code review is the single most effective practice you can introduce to share knowledge around, and build the quality and consistency of your team’s work. But, if you’re not doing it, here’s my suggestion for getting started:
Every pull request gets a +1 from someone else.
That’s all — it’s very light-weight and easy. Just ask someone to have a quick look over your code before it goes into master.
At Twitter, every commit gets a code review. We do a lot of reviewing, so small efficiency and effectiveness improvements make a big difference. Over time we learned some things:
Don’t review for more than hour 1
Keep reviews smaller than ~400 lines 2
Code review your own code first 2
After an hour, and above roughly 400 lines, your ability to detect issues in a code review starts to decrease. So review little and often. The gaps around lunch, standup and before you head home are ideal. And remember, if someone’s put code up for a review, that review is blocking them doing other work. It’s your job to unblock them.
On TweetDeck, we actually try to keep reviews under 250 lines. It doesn’t sound like much, but this constraint applies pressure to make smaller, incremental changes. This makes breakages easier to detect and roll back, and leads to a very natural feature development process that encourages learning and iteration.
But the most important thing I’ve learned personally is that reviewing my own code is the best way to spot issues. I try to approach my own reviews the way I approach my team’s: with fresh, critical eyes, after a break, using a dedicated code review tool.
It’s amazing what you can spot when you put a new in a new interface around code you’ve been staring at for hours!
And yes, this list features science. The data backs up these conclusions, and if you’d like to learn more about scientific approaches to software engineering then I recommend you buy Making Software: What Really Works, and Why We Believe It. It’s ace.
For more dedicated information sharing, we’ve introduced regular seminars for everyone who works on a specific area or technology. It works like this: a team-member shares or teaches something to everyone else, and next time it’s someone else’s turn. Giving everyone a chance to speak, and encouraging a wide range of topics, is starting to produce great results.
If you’d like to run a seminar, one thing you could try to get started: run a point at the thing you least understand in our architecture session — thanks to James for this idea. And guess what… your onboarding architecture diagrams will help (and benefit from) this!
More, please!
There’s a few ideas here to get you started, but there are even more in a talk I gave this year called Frontend Archaeology, including a look at optimising for confidence with front-end operations.
And finally, thanks to Amy for proof reading this and to Passy for feedback on the original talk.
Dunsmore et al. 2000. Object-Oriented Inspection in the Face of Delocalisation. Beverly, MA: SmartBear Software. ↩
Cohen, Jason. 2006. Best Kept Secrets of Peer Code Review. Proceedings of the 22nd ICSE 2000: 467-476. ↩ ↩",2016,Tom Ashworth,tomashworth,2016-12-18T00:00:00+00:00,https://24ways.org/2016/new-tricks-for-an-old-dog/,code
191,CSS Animations,"Friend: You should learn how to write CSS!
Me: …
Friend: CSS; Cascading Style Sheets. If you’re serious about web design, that’s the next thing you should learn.
Me: What’s wrong with tags?
That was 8 years ago. Thanks to the hard work of Jeffrey, Andy, Andy, Cameron, Colly, Dan and many others, learning how to decently markup a website and write lightweight stylesheets was surprisingly easy. They made it so easy even a complete idiot (OH HAI) was able to quickly master it.
And then… nothing. For a long time, it seemed like there wasn’t happening anything in the land of CSS, time stood still. Once you knew the basics, there wasn’t anything new to keep up with. It looked like a great band split, but people just kept re-releasing their music in various “Best Of!” or “Remastered!” albums.
Fast forward a couple of years to late 2006. On the official WebKit blog Surfin’ Safari, there’s an article about something called CSS animations. Great new stuff to play with, but only supported by nightly builds (read: very, very beta) of WebKit. In the following months, they release other goodies, like CSS gradients, CSS reflections, CSS masks, and even more CSS animation sexiness. Whoa, looks like the band got back together, found their second youth, and went into overdrive! The problem was that if you wanted to listen to their new albums, you had to own some kind of new high-tech player no one on earth (besides some early adopters) owned.
Back in the time machine. It is now late 2009, close to Christmas. Things have changed. Browsers supporting these new toys are widely available left and right. Even non-techies are using these advanced browsers to surf the web on a daily basis!
Epic win? Almost, but at least this gives us enough reason to start learning how we could use all this new CSS voodoo. On Monday, Natalie Downe showed you a good tutorial on Going Nuts with CSS Transitions. Today, I’m taking it one step further…
Howto: A basic spinner
No matter how fast internet tubes or servers are, we’ll always need spinners to indicate something’s happening behind the scenes. Up until now, people would go to some site, pick one of the available templates, customize their foreground and background colors, and download a beautiful GIF image.
There are some downsides to this though:
It’s only _semi_-transparent: If you change your mind and pick a slightly different background color, you need to go back to the site, set all the parameters again, and replace your current image. There isn’t even a way to pick an image or gradient as background.
Limited number of frames, probable to keep the file-size as small as possible (don’t forget this thing needs to be loaded before whatever process is finished in the background), and you don’t have that 24 frames per second smoothness.
This is just too fucking easy. As a front-end code geek, there must be a “cooler” way to do this!
What do we need to make a spinner with CSS animations? One image, and one element on our webpage we can hook on to. Yes, that’s it. I created a simple transparent PNG that looks it might be a spinner, and for the element on the page, I wrote this piece of genius HTML:
Please wait while we do what we do best.
Looks semantic enough to me! Here’s the basic HTML I’m using to position the element in the center of the screen, and make the text inside it disappear:
#spinner {
position: absolute;
top: 50%;
left: 50%;
margin: -100px 0 0 -100px;
height: 200px;
width: 200px;
text-indent: 250px;
white-space: nowrap;
overflow: hidden;
}
Cool, but now we don’t see anything. Let’s pull rabbit number one out of the hat: -webkit-mask-image (accompanied by the previously mentioned transparent PNG image):
#spinner {
...
-webkit-mask-image: url(../img/spinner.png);
}
By now you should be feeling like a magician already. Oh, wait, we still have a blank screen, looks like we left something in the hat (tip: not rabbit droppings):
#spinner {
...
-webkit-mask-image: url(../img/spinner.png);
background-color: #000;
}
Nice! What we’ve done right here is telling the element to clip onto the PNG. It’s a lot like clipping layers in Photoshop. So, spinners, they move, right? Into the hat again, and look what we pull out this time: CSS animations!
#spinner {
...
-webkit-mask-image: url(../img/spinner.png);
background-color: #000;
-webkit-animation-name: spinnerRotate;
-webkit-animation-duration: 2s;
-webkit-animation-iteration-count: infinite;
-webkit-animation-timing-function: linear;
}
Some explanation:
-webkit-animation-name: Name of the animation we’ll be defining later.
-webkit-animation-duration: The timespan of the animation.
-webkit-animation-iteration-count: Repeat once, a defined number of times or infinitely?
-webkit-animation-timing-function: Linear is the one you’ll be using mostly. Other options are ease-in, ease-out, ease-in-out…
Let’s define spinnerRotate:
@-webkit-keyframes spinnerRotate {
from {
-webkit-transform:rotate(0deg);
}
to {
-webkit-transform:rotate(360deg);
}
}
En Anglais: Rotate #spinner starting at 0 degrees, ending at 360 degrees, over a timespan of 2 seconds, at a constant speed, and keep repeating this animation forever.
That’s it! See it in action on the demo page.
Note: these examples only work when you’re using a WebKit-based browser like Safari, Mobile Safari or Google Chrome. I’m confident though that Mozilla and Opera will try their very best catching up with all this new CSS goodness soon.
When looking at this example, you see the possibilities are endless. Another advantage is you can change the look of it entirely by only changing a couple of lines of CSS, instead of re-creating and re-downloading the image from some website smelling like web 2.0 gone bad. I made another demo that shows how great it is to be able to change background and foreground colors (even on the fly!).
So there you have it, a smoothly animated, fully transparent and completely customizable spinner. Cool? I think so. (Ladies?)
But you can do a lot more with CSS animations than just create pretty spinners. Since I was fooling around with it anyway, I decided to test how far you can push this, space is the final limit, right?
Conclusion
CSS has never been more exciting than it is right now. I’m even prepared to say CSS is “cool” again, both for the more experienced front-end developers as for the new designers discovering CSS every day now.
But…
Remember when Javascript became popular? Remember when Flash became popular? Every time we’re been given new toys, some people aren’t ashamed to use it in a way you can barely call constructive. I’m thinking of Geocities websites, loaded with glowing blocks of text, moving images, bad color usage… In the wise words of Stan Lee: With great power there must also come great responsibility! A sprinkle of CSS animations is better than a bucket load. Apply with care.",2009,Tim Van Damme,timvandamme,2009-12-15T00:00:00+00:00,https://24ways.org/2009/css-animations/,code
202,Design Systems and CSS Grid,"Recently, my client has been looking at creating a few new marketing pages for their website. They currently have a design system in place but they’re looking to push this forward into 2018 with some small and possibly some larger changes.
To start with we are creating a couple of new marketing pages. As well as making use of existing components within the design systems component library there are a couple of new components. Looking at the first couple of sketch files I felt this would be a great opportunity to use CSS Grid, to me the newer components need to be laid out on that page and grid would help with this perfectly.
As well as this layout of the new components and the text within it, imagery would be used that breaks out of the grid and pushes itself into the spaces where the text is aligned.
The existing grid system
When the site was rebuilt in 2015 the team decided to make use of Sass and Susy, a “lightweight grid-layout engine using Sass”. It was built separating the grid system from the components that would be laid out on the page with a container, a row, an optional column, and a block.
To make use of the grid system on a page for a component that would take the full width of the row you would have to write something like this:
Using a grid system similar to this can easily create quite the tag soup. It could fill the HTML full of divs that may become complex to understand and difficult to edit.
Although there is this reliance on several
s to lay out the components on a page it does allow a tidy way to place the component code within that page. It abstracts the layout of the page to its own code, its own system, so the components can ‘fit’ where needed.
The requirements of the new grid system
Moving forward I set myself some goals for what I’d like to have achieved in this new grid system:
It needs to behave like the existing grid systems
We are not ripping up the existing grid system, it would be too much work, for now, to retrofit all of the existing components to work in a grid that has a different amount of columns, and spacing at various viewport widths.
Allow full-width components
Currently the grid system is a 14 column grid that becomes centred on the page when viewport is wide enough. We have, in the past, written some CSS that would allow for a full-width component, but his had always felt like a hack. We want the option to have a full-width element as part of the new grid system, not something that needs CSS to fight against.
Less of a tag soup
Ideally we want to end up writing less HTML to layout the page. Although the existing system can be quite clear as to what each element is doing, it can also become a little laborious in working out what each grid row or block is doing where.
I would like to move the layout logic to CSS as much as is possible, potentially creating some utility classes or additional ‘layout classes’ for the components.
Easier for people to use and author
With many people using the existing design systems codebase we need to create a new grid system that is as easy or easier to use than the existing one. I think and hope this would be helped by removing as many
s as needed and would require new documentation and examples, and potentially some initial training.
Separating layout from style
There still needs to be a separation of layout from the styles for the component. To allow for the component itself to be placed wherever needed in the page we need to make sure that the CSS for the layout is a separate entity to the CSS for that styling.
With these base requirements I took to CodePen and started working on some throwaway code to get started.
Making the new grid(s)
The Full-Width Grid
To start with I created a grid that had three columns, one for the left, one for the middle, and one for the right. This would give the full-width option to components.
Thankfully, one of Rachel Andrew’s many articles on Grid discussed this exact requirement of the new grid system to break out with Grid.
I took some of the code in the examples and edited to make grid we needed.
.container {
display: grid;
grid-template-columns:
[full-start]
minmax(.75em, 1fr)
[main-start]
minmax(0, 1008px)
[main-end]
minmax(.75em, 1fr)
[full-end];
}
We are declaring a grid, we have four grid column lines which we name and we define how the three columns they create react to the viewport width. We have a left and right column that have a minimum of 12px, and a central column with a maximum width of 1008px.
Both left and right columns fill up any additional space if the viewport is wider that 1032px wide. We are also not declaring any gutters to this grid, the left and right columns would act as gutters at smaller viewports.
At this point I noticed that older versions of Sass cannot parse the brackets in this code. To combat this I used Sass’ unquote method to wrap around the value of the grid-template-column.
.container {
display: grid;
grid-template-columns:
unquote(""
[full-start]
minmax(.75em, 1fr)
[main-start]
minmax(0, 1008px)
[main-end]
minmax(.75em, 1fr)
[full-end]
"");
}
The existing codebase makes use of Sass variables, mixins and functions so to remove that would be a problem, but luckily the version of Sass used is up-to-date (note: example CodePens will be using CSS).
The initial full-width grid displays on a webpage as below:
The 14 column grid
I decided to work out the 14 column grid as a separate prototype before working out how it would fit within the full-width grid. This grid is very similar to the 12 column grids that have been used in web design. Here we need 14 columns with a gutter between each one.
Along with the many other resources on Grid, Mozilla’s MDN site had a page on common layouts using CSS Grid. This gave me the perfect CSS I needed to create my grid and I edited it as required:
.inner {
display: grid;
grid-template-columns: repeat(14, [col-start] 1fr);
grid-gap: .75em;
}
We, again, are declaring a grid, and we are splitting up the available space by creating 14 columns with 1 fr-unit and giving each one a starting line named col-start.
This grid would display on web page as below:
Bringing the grids together
Now that we have got the two grids we need to help fulfil our requirements we need to put them together so that they are actually we we need.
The subgrid
There is no subgrid in CSS, yet. To workaround this for the new grid system we could nest the 14 column grid inside the full-width grid.
In the HTML we nest the 14 column inner grid inside the full-width container.
So that the inner knows where to be laid out within the container we tell it what column to start and end with, with this code it would be the start and end of the main column.
.inner {
display: grid;
grid-column: main-start / main-end;
grid-template-columns: repeat(14, [col-start] 1fr);
grid-gap: .75em;
}
The CSS for the container remains unchanged.
This works, but we have added another div to our HTML. One of our requirements is to try and remove the potential for tag soup.
The faux subgrid subgrid
I wanted to see if it would be possible to place the CSS for the 14 column grid within the CSS for the full-width grid. I replaced the CSS for the main grid and added the grid-column-gap to the .container.
.container {
display: grid;
grid-gap: .75em;
grid-template-columns:
[full-start]
minmax(.75em, 1fr)
[main-start]
repeat(14, [col-start] 1fr)
[main-end]
minmax(.75em, 1fr)
[full-end];
}
What this gave me was a 16 column grid. I was unable to find a way to tell the main grid, the grid betwixt main-start and main-end to be a maximum of 1008px as required.
I trawled the internet to find if it was possible to create our main requirement, a 14 column grid which also allows for full-width components. I found that we could not reverse minmax to minmax(1fr, 72px) as 1fr is not allowed as a minimum if there is a maximum. I tried working out if we could make the min larger than its max but in minmax it would be ignored.
I was struggling, I was hoping for a cleaner version of the grid system we currently use but getting to the point where needing that extra
would be a necessity.
At 3 in the morning, when I was failing to get to sleep, my mind happened upon an question: “Could you use calc?”
At some point I drifted back to sleep so the next day I set upon seeing if this was possible. I knew that the maximum width of the central grid needed to be 1008px. The left and right columns needed to be however many pixels were left in the viewport divided by 2. In CSS it looked like I would need to use calc twice. The first time to takeaway 1008px from 100% of the viewport width and the second to divide that result by 2.
calc(calc(100% - 1008px) / 2)
The CSS above was part of the value that I would need to include in the declaration for the grid.
.container {
display: grid;
grid-gap: .75em;
grid-template-columns:
[full-start]
minmax(calc(calc(100% - 1008px) / 2), 1fr)
[main-start]
repeat(14, [col-start] 1fr)
[main-end]
minmax(calc(calc(100% - 1008px) / 2), 1fr)
[full-end];
}
We have created the grid required. A full-width grid, with a central 14 column grid, using fewer
elements.
See the Pen Design Systems and CSS Grid, 6 by Stuart Robson (@sturobson) on CodePen.
Success!
Progressive enhancement
Now that we have created the grid system required we need to back-track a little.
Not all browsers support Grid, over the last 9 months or so this has gotten a lot better. However there will still be browsers that visit that potentially won’t have support. The effort required to make the grid system fall back for these browsers depends on your product or sites browser support.
To determine if we will be using Grid or not for a browser we will make use of feature queries. This would mean that any version of Internet Explorer will not get Grid, as well as some mobile browsers and older versions of other browsers.
@supports (display: grid) {
/* Styles for browsers that support Grid */
}
If a browser does not pass the requirements for @supports we will fallback to using flexbox where possible, and if that is not supported we are happy for the page to be laid out in one column.
A website doesn’t have to look the same in every browser after all.
A responsive grid
We started with the big picture, how the grid would be at a large viewport and the grid system we have created gets a little silly when the viewport gets smaller.
At smaller viewports we have a single column layout where every item of content, every component stacks atop each other. We don’t start to define a grid before we the viewport gets to 700px wide. At this point we have an 8 column grid and if the viewport gets to 1100px or wider we have our 14 column grid.
/*
* to start with there is no 'grid' just a single column
*/
.container {
padding: 0 .75em;
}
/*
* when we get to 700px we create an 8 column grid with
* a left and right area to breakout of the grid.
*/
@media (min-width: 700px) {
.container {
display: grid;
grid-gap: .75em;
grid-template-columns:
[full-start]
minmax(calc(calc(100% - 1008px) / 2), 1fr)
[main-start]
repeat(8, [col-start] 1fr)
[main-end]
minmax(calc(calc(100% - 1008px) / 2), 1fr)
[full-end];
padding: 0;
}
}
/*
* when we get to 1100px we create an 14 column grid with
* a left and right area to breakout of the grid.
*/
@media (min-width: 1100px) {
.container {
grid-template-columns:
[full-start]
minmax(calc(calc(100% - 1008px) / 2), 1fr)
[main-start]
repeat(14, [col-start] 1fr)
[main-end]
minmax(calc(calc(100% - 1008px) / 2), 1fr)
[full-end];
}
}
Being explicit in creating this there is some repetition that we could avoid, we will define the number of columns for the inner grid by using a Sass variable or CSS custom properties (more commonly termed as CSS variables).
Let’s use CSS custom properties. We need to declare the variable first by adding it to our stylesheet.
:root {
--inner-grid-columns: 8;
}
We then need to edit a few more lines. First make use of the variable for this line.
repeat(8, [col-start] 1fr)
/* replace with */
repeat(var(--inner-grid-columns), [col-start] 1fr)
Then at the 1100px breakpoint we would only need to change the value of the —inner-grid-columns value.
@media (min-width: 1100px) {
.container {
grid-template-columns:
[full-start]
minmax(calc(calc(100% - 1008px) / 2), 1fr)
[main-start]
repeat(14, [col-start] 1fr)
[main-end]
minmax(calc(calc(100% - 1008px) / 2), 1fr)
[full-end];
}
}
/* replace with */
@media (min-width: 1100px) {
.container {
--inner-grid-columns: 14;
}
}
See the Pen Design Systems and CSS Grid, 8 by Stuart Robson (@sturobson) on CodePen.
The final grid system
We have finally created our new grid for the design system. It stays true to the existing grid in place, adds the ability to break-out of the grid, removes a
that could have been needed for the nested 14 column grid.
We can move on to the new component.
Creating a new component
Back to the new components we are needing to create.
To me there are two components one of which is a slight variant of the first. This component contains a title, subtitle, a paragraph (potentially paragraphs) of content, a list, and a call to action.
To start with we should write the HTML for the component, something like this:
To place the component on the existing grid is fine, but as child elements are not affected by the container grid we need to define another grid for the features component.
As the grid doesn’t get invoked until 700px it is possible to negate the need for a media query.
.features {
grid-column: col-start 1 / span 6;
}
@supports (display: grid) {
@media (min-width: 1100px) {
.features {
grid-column-end: 9;
}
}
}
We can also avoid duplication of declarations by making use of the grid-column shorthand and longhand. We need to write a little more CSS for the variant component, the one that will sit on the right side of the page too.
.features:nth-of-type(even) {
grid-column-start: 4;
grid-row: 2;
}
@supports (display: grid) {
@media (min-width: 1100px) {
.features:nth-of-type(even) {
grid-column-start: 9;
grid-column-end: 16;
}
}
}
We cannot place the items within features on the container grid as they are not direct children. To make this work we have to define a grid for the features component.
We can do this by defining the grid at the first breakpoint of 700px making use of CSS custom properties again to define how many columns there will need to be.
.features {
grid-column: col-start 1 / span 6;
--features-grid-columns: 5;
}
@supports (display: grid) {
@media (min-width: 700px) {
.features {
display: grid;
grid-gap: .75em;
grid-template-columns: repeat(var(--features-grid-columns), [col-start] 1fr);
}
}
}
@supports (display: grid) {
@media (min-width: 1100px) {
.features {
grid-column-end: 9;
--features-grid-columns: 7;
}
}
}
See the Pen Design Systems and CSS Grid, 10 by Stuart Robson (@sturobson) on CodePen.
Laying out the parts
Looking at the spec and reading several articles I feel there are two ways that I could layout the text of this component on the grid.
We could use the grid-column shorthand that incorporates grid-column-start and grid-column-end or we can make use of grid-template-areas.
grid-template-areas allow for a nice visual way of representing how the parts of the component would be laid out. We can take the the mock of the features on the grid and represent them in text in our CSS.
Within the .features rule we can add the relevant grid-template-areas value to represent the above.
.features {
display: grid;
grid-template-columns: repeat(var(--features-grid-columns), [col-start] 1fr);
grid-template-areas:
"". title title title title title title""
"". subtitle subtitle subtitle subtitle subtitle . ""
"". content content content content . . ""
"". list list list . . . ""
"". . . . link link link "";
}
In order to make the variant of the component we would have to create the grid-template-areas for that component too.
We then need to tell each element of the component in what grid-area it should be placed within the grid.
.features__title { grid-area: title; }
.features__subtitle { grid-area: subtitle; }
.features__content { grid-area: content; }
.features__list { grid-area: list; }
.features__link { grid-area: link; }
See the Pen Design Systems and CSS Grid, 12 by Stuart Robson (@sturobson) on CodePen.
The other way would be to use the grid-column shorthand and the grid-column-start and grid-column-end we have used previously.
.features .features__title {
grid-column: col-start 2 / span 6;
}
.features .features__subtitle {
grid-column: col-start 2 / span 5;
}
.features .features__content {
grid-column: col-start 2 / span 4;
}
.features .features__list {
grid-column: col-start 2 / span 4;
}
.features .features__link {
grid-column: col-start 5 / span 3;
}
For the variant of the component we can use the grid-column-start property as it will inherit the span defined in the grid-column shorthand.
.features:nth-of-type(even) .features__title {
grid-column-start: col-start 1;
}
.features:nth-of-type(even) .features__subtitle {
grid-column-start: col-start 1;
}
.features:nth-of-type(even) .features__content {
grid-column-start: col-start 3;
}
.features:nth-of-type(even) .features__list {
grid-column-start: col-start 3;
}
.features:nth-of-type(even) .features__link {
grid-column-start: col-start 1;
}
See the Pen Design Systems and CSS Grid, 14 by Stuart Robson (@sturobson) on CodePen.
I think, for now, we will go with using grid-column properties rather than grid-template-areas. The repetition needed for creating the variant feels too much where we can change the grid-column-start instead, keeping the components elements layout properties tied a little closer to the elements rather than the grid.
Some additional decisions
The current component library has existing styles for titles, subtitles, lists, paragraphs of text and calls to action. These are name-spaced so that they shouldn’t clash with any other components. Looking forward there will be a chance that other products adopt the component library, but they may bring their own styles for titles, subtitles, etc.
One way that we could write our code now for that near future possibility is to make sure our classes are working hard. Using class-attribute selectors we can target part of the class attributes that we know the elements in the component will have using *=.
.features [class*=""title""] {
grid-column: col-start 2 / span 6;
}
.features [class*=""subtitle""] {
grid-column: col-start 2 / span 5;
}
.features [class*=""content""] {
grid-column: col-start 2 / span 4;
}
.features [class*=""list""] {
grid-column: col-start 2 / span 4;
}
.features [class*=""link""] {
grid-column: col-start 5 / span 3;
}
See the Pen Design Systems and CSS Grid, 15 by Stuart Robson (@sturobson) on CodePen.
Although the component we have created have a title, subtitle, paragraphs, a list, and a call to action there may be a time where one ore more of these is not required or available. One thing I found out is that if the element doesn’t exist then grid will not create space for it. This may be obvious, but it can be really helpful in making a nice malleable component.
We have only looked at columns, as existing components have their own spacing for the vertical rhythm of the page we don’t really want to have them take up equal space in the component and just take up the space as needed. We can do this by adding grid-auto-rows: min-content; to our .features. This is useful if you also need your component to take up a height that is more than the component itself.
The grid of the future
From prototyping this new grid and components in CSS Grid, I’ve found it a fantastic way to reimagine how we can create a layout or grid system for our sites. It gives us options to create the same layouts in differing ways that could suit a project and its needs.
It allows us to carry on – if we choose to – using a
-based grid but swapping out floats for CSS Grid or to tie it to our components so they have specific places to go depending on what component is being used. Or we could have several ‘grid components’ in our design system that we could use to layout various components throughout a page.
If you find yourself tasked with creating some new components for your design system try it. If you are starting from scratch I believe you really should start with CSS Grid for your layout.
It really feels like the possibilities are endless in terms of layout for the web.
Resources
Here are just a few resources I have pawed over these last few weeks whilst getting acquainted with CSS Grid.
A collection of CodePens from this article
Grid by Example from Rachel Andrew
A Complete Guide to CSS Grid on Codrops from Hui Jing Chen
Rachel Andrew’s Blog Archive tagged: cssgrid
CSS Grid Layout Examples
MDN’s CSS Grid Layout
A Complete Guide to Grid from CSS-Tricks
CSS Grid Layout Module Level 1 Specification",2017,Stuart Robson,stuartrobson,2017-12-12T00:00:00+00:00,https://24ways.org/2017/design-systems-and-css-grid/,code
157,Capturing Caps Lock,"One of the more annoying aspects of having to remember passwords (along with having to remember loads of them) is that if you’ve got Caps Lock turned on accidentally when you type one in, it won’t work, and you won’t know why. Most desktop computers alert you in some way if you’re trying to enter your password to log on and you’ve enabled Caps Lock; there’s no reason why the web can’t do the same. What we want is a warning – maybe the user wants Caps Lock on, because maybe their password is in capitals – rather than something that interrupts what they’re doing. Something subtle.
But that doesn’t answer the question of how to do it. Sadly, there’s no way of actually detecting whether Caps Lock is on directly. However, there’s a simple work-around; if the user presses a key, and it’s a capital letter, and they don’t have the Shift key depressed, why then they must have Caps Lock on! Simple.
DOM scripting allows your code to be notified when a key is pressed in an element; when the key is pressed, you get the ASCII code for that key. Capital letters, A to Z, have ASCII codes 65 to 90. So, the code would look something like:
on a key press
if the ASCII code for the key is between 65 and 90 *and* if shift is pressed
warn the user that they have Caps Lock on, but let them carry on
end if
end keypress
The actual JavaScript for this is more complicated, because both event handling and keypress information differ across browsers. Your event handling functions are passed an event object, except in Internet Explorer where you use the global event object; the event object has a which parameter containing the ASCII code for the key pressed, except in Internet Explorer where the event object has a keyCode parameter; some browsers store whether the shift key is pressed in a shiftKey parameter and some in a modifiers parameter. All this boils down to code that looks something like this:
keypress: function(e) {
var ev = e ? e : window.event;
if (!ev) {
return;
}
var targ = ev.target ? ev.target : ev.srcElement;
// get key pressed
var which = -1;
if (ev.which) {
which = ev.which;
} else if (ev.keyCode) {
which = ev.keyCode;
}
// get shift status
var shift_status = false;
if (ev.shiftKey) {
shift_status = ev.shiftKey;
} else if (ev.modifiers) {
shift_status = !!(ev.modifiers & 4);
}
// At this point, you have the ASCII code in “which”,
// and shift_status is true if the shift key is pressed
}
Then it’s just a check to see if the ASCII code is between 65 and 90 and the shift key is pressed. (You also need to do the same work if the ASCII code is between 97 (a) and 122 (z) and the shift key is not pressed, because shifted letters are lower-case if Caps Lock is on.)
if (((which >= 65 && which <= 90) && !shift_status) ||
((which >= 97 && which <= 122) && shift_status)) {
// uppercase, no shift key
/* SHOW THE WARNING HERE */
} else {
/* HIDE THE WARNING HERE */
}
The warning can be implemented in many different ways: highlight the password field that the user is typing into, show a tooltip, display text next to the field. For simplicity, this code shows the warning as a previously created image, with appropriate alt text. Showing the warning means creating a new tag with DOM scripting, dropping it into the page, and positioning it so that it’s next to the appropriate field. The image looks like this:
You know the position of the field the user is typing into (from its offsetTop and offsetLeft properties) and how wide it is (from its offsetWidth properties), so use createElement to make the new img element, and then absolutely position it with style properties so that it appears in the appropriate place (near to the text field).
The image is a transparent PNG with an alpha channel, so that the drop shadow appears nicely over whatever else is on the page. Because Internet Explorer version 6 and below doesn’t handle transparent PNGs correctly, you need to use the AlphaImageLoader technique to make the image appear correctly.
newimage = document.createElement('img');
newimage.src = ""http://farm3.static.flickr.com/2145/2067574980_3ddd405905_o_d.png"";
newimage.style.position = ""absolute"";
newimage.style.top = (targ.offsetTop - 73) + ""px"";
newimage.style.left = (targ.offsetLeft + targ.offsetWidth - 5) + ""px"";
newimage.style.zIndex = ""999"";
newimage.setAttribute(""alt"", ""Warning: Caps Lock is on"");
if (newimage.runtimeStyle) {
// PNG transparency for IE
newimage.runtimeStyle.filter += ""progid:DXImageTransform.Microsoft.AlphaImageLoader(src='http://farm3.static.flickr.com/2145/2067574980_3ddd405905_o_d.png',sizingMethod='scale')"";
}
document.body.appendChild(newimage);
Note that the alt text on the image is also correctly set. Next, all these parts need to be pulled together. On page load, identify all the password fields on the page, and attach a keypress handler to each. (This only needs to be done for password fields because the user can see if Caps Lock is on in ordinary text fields.)
var inps = document.getElementsByTagName(""input"");
for (var i=0, l=inps.length; i
The “create an image” code from above should only be run if the image is not already showing, so instead of creating a newimage object, create the image and attach it to the password field so that it can be checked for later (and not shown if it’s already showing). For safety, all the code should be wrapped up in its own object, so that its functions don’t collide with anyone else’s functions. So, create a single object called capslock and make all the functions be named methods of the object:
var capslock = {
...
keypress: function(e) {
}
...
}
Also, the “create an image” code is saved into its own named function, show_warning(), and the converse “remove the image” code into hide_warning(). This has the advantage that developers can include the JavaScript library that has been written here, but override what actually happens with their own code, using something like:
And that’s all. Simply include the JavaScript library in your pages, override what happens on a warning if that’s more appropriate for what you’re doing, and that’s all you need.
See the script in action.",2007,Stuart Langridge,stuartlangridge,2007-12-04T00:00:00+00:00,https://24ways.org/2007/capturing-caps-lock/,code
201,Lint the Web Forward With Sonarwhal,"Years ago, when I was in a senior in college, much of my web development courses focused on two things: the basics like HTML and CSS (and boy, do I mean basic), and Adobe Flash. I spent many nights writing ActionScript 3.0 to build interactions for the websites that I would add to my portfolio. A few months after graduating, I built one website in Flash for a client, then never again. Flash was dying, and it became obsolete in my résumé and portfolio.
That was my first lesson in the speed at which things change in technology, and what a daunting realization that was as a new graduate looking to enter the professional world. Now, seven years later, I work on the Microsoft Edge team where I help design and build a tool that would have lessened my early career anxieties: sonarwhal.
Sonarwhal is a linting tool, built by and for the web community. The code is open source and lives under the JS Foundation. It helps web developers and designers like me keep up with the constant change in technology while simultaneously teaching how to code better websites.
Introducing sonarwhal’s mascot Nellie
Good web development is hard. It is more than HTML, CSS, and JavaScript: developers are expected to have a grasp of accessibility, performance, security, emerging standards, and more, all while refreshing this knowledge every few months as the web evolves. It’s a lot to keep track of.
Web development is hard
Staying up-to-date on all this knowledge is one of the driving forces for developing this scanning tool. Whether you are just starting out, are a student, or you have over a decade of experience, the sonarwhal team wants to help you build better websites for all browsers.
Currently sonarwhal checks for best practices in five categories: Accessibility, Interoperability, Performance, PWAs, and Security. Each check is called a “rule”. You can configure them and even create your own rules if you need to follow some specific guidelines for your project (e.g. validate analytics attributes, title format of pages, etc.).
You can use sonarwhal in two ways:
An online version, that provides a quick and easy way to scan any public website.
A command line tool, if you want more control over the configuration, or want to integrate it into your development flow.
The Online Scanner
The online version offers a streamlined way to scan a website; just enter a URL and you will get a web page of scan results with a permalink that you can share and revisit at any time.
The online version of sonarwal
When my team works on a new rule, we spend the bulk of our time carefully researching each subject, finding sources, and documenting it rather than writing the rule’s code. Not only is it important that we get you the right results, but we also want you to understand why something is failing. Next to each failing rule you’ll find a link to its detailed documentation, explaining why you should care about it, what exactly we are testing, examples that pass and examples that don’t, and useful links to even more in-depth documentation if you are interested in the subject.
We hope that between reading the documentation and continued use of sonarwhal, developers can stay on top of best practices. As devs continue to build sites and identify recurring issues that appear in their results, they will hopefully start to automatically include those missing elements or fix those pieces of code that are producing errors. This also isn’t a one-way communication: the documentation is not only available on the sonarwhal site, but also on GitHub for editing so you can help us make it even better!
A results report
The current configuration for the online scanner is very strict, so it might hurt your feelings (it did when I first tested it on my personal website). But you can configure sonarwhal to any level of strictness as well as customize the command line tool to your needs!
Sonarwhal’s CLI
The CLI gives you full control of sonarwhal: what rules to use, tweaks to them, domains that are out of your control, and so on. You will need the latest node LTS (v8) or Stable (v9) and your favorite package manager, such as npm:
npm install -g sonarwhal
You can now run sonarwhal from anywhere via:
sonarwhal https://example.com
Using the CLI
The configuration is done via a .sonarwhalrc file. When analyzing a site, if no file is available, you will be prompted to answer a series of questions:
What connector do you want to use? Connectors are what sonarwhal uses to access a website and gather all the information about the requests, resources, HTML, etc. Currently it supports jsdom, Microsoft Edge, and Google Chrome.
What formatter? This is how you want to see the results: summary, stylish, etc. Make sure to look at the full list. Some are concise for, perfect for a quick build assessment, while others are more verbose and informative.
Do you want to use the recommended rules configuration? Rules are the things we are validating. Unless you’ve read the documentation and know what you are doing, first timers should probably use the recommended configuration.
What browsers are you targeting? One of the best features of sonarwhal is that rules can adapt their feedback depending on your targeted browsers, suggesting to add or remove things. For example, the rule “Highest Document Mode” will tell you to add the “X-UA-Compatible” header if IE10 or lower is targeted or remove if the opposite is true.
sonarwhal configuration generator questions
Once you answer all these questions the scan will start and you will have a .sonarwhalrc file similar to the following:
{
""connector"": {
""name"": ""jsdom"",
""options"": {
""waitFor"": 1000
}
},
""formatters"": ""stylish"",
""rulesTimeout"": 120000,
""rules"": {
""apple-touch-icons"": ""error"",
""axe"": ""error"",
""content-type"": ""error"",
""disown-opener"": ""error"",
""highest-available-document-mode"": ""error"",
""validate-set-cookie-header"": ""warning"",
// ...
}
}
You should see the scan initiate in the command line and within a few seconds the results should start to appear. Remember, the scan results will look different depending on which formatter you selected so try each one out to see which one you like best.
sonarwhal results on my website and hurting my feelings 💔
Now that you have a list of errors, you can get to work improving the site! Note though, that when you scan your website, it scans all the resources on that page and if you’ve added something like analytics or fonts hosted elsewhere, you are unable to change those files. You can configure the CLI to ignore files from certain domains so that you are only getting results for files you are in control of.
The documentation should give enough guidance on how to fix the errors, but if it’s insufficient, please help us and suggest edits or contribute back to it. This is a community effort and chances are someone else will have the same question as you.
When I scanned both my websites, sonarwhal alerted me to not having an Apple Touch Icon. If I search on the web as opposed to using the sonarwhal documentation, the first top 3 results give me outdated information: I need to include many different icon sizes. I don’t need to include all the different size icons that target different devices. Declaring one icon sized 180px x 180px will provide a large enough icon for devices and it will scale down as appropriate for people on older devices.
The information at the top of the search results isn’t always the correct answer to an issue and we don’t want you to have to search through outdated documentation. As sonarwhal’s capabilities expand, the goal is for it to be the one stop shop for helping preflight your website.
The journey up until now and looking forward
On the Microsoft Edge team, we’re passionate about empowering developers to build great websites. Every day we see so many sites come through our issue tracker. (Thanks for filing those bugs, they help us make Microsoft Edge better and better!) Some issues we see over and over are honest mistakes or outdated ‘best practices’ that could be avoided, so we built this tool to help everyone help make the web a better place.
When we decided to create sonarwhal, we wanted to create a tool that would help developers write better and more up-to-date code for their websites. We want sonarwhal to be useful to anyone so, early on, we defined three guiding principles we’ve used along the way:
Community Driven. We build for the community’s best interests. The web belongs to everyone and this project should too. Not only is it open source, we’ve also donated it to the JS Foundation and have an inclusive governance model that welcomes the collaboration of anyone, individual or company.
User Centric. We want to put the user at the center, making sonarwhal configurable for your needs and easy to use no matter what your skill level is.
Collaborative. We didn’t want to reinvent the wheel, so we collaborated with existing tools and services that help developers build for the web. Some examples are aXe, snyk.io, Cloudinary, etc.
This is just the beginning and we still have lots to do. We’re hard at work on a backlog of exciting features for future releases, such as:
New rules for a variety of areas like performance, accessibility, security, progressive web apps, and more.
A plug-in for Visual Studio Code: we want sonarwhal to help you write better websites, and what better moment than when you are in your editor.
Configuration options for the online service: as we fine tune the infrastructure, the rule configuration for our scanner is locked, but we look forward to adding CLI customization options here in the near future.
This is a tool for the web community by the web community so if you are excited about sonarwhal, making a better web, and want to contribute, we have a few issues where you might be able to help. Also, don’t forget to check the rest of the sonarwhal GitHub organization. PRs are always welcome and appreciated!
Let us know what you think about the scanner at @NarwhalNellie on Twitter and we hope you’ll help us lint the web forward!",2017,Stephanie Drescher,stephaniedrescher,2017-12-02T00:00:00+00:00,https://24ways.org/2017/lint-the-web-forward-with-sonarwhal/,code
168,Unobtrusively Mapping Microformats with jQuery,"Microformats are everywhere. You can’t shake an electronic stick these days without accidentally poking a microformat-enabled site, and many developers use microformats as a matter of course. And why not? After all, why invent your own class names when you can re-use pre-defined ones that give your site extra functionality for free?
Nevertheless, while it’s good to know that users of tools such as Tails and Operator will derive added value from your shiny semantics, it’s nice to be able to reuse that effort in your own code.
We’re going to build a map of some of my favourite restaurants in Brighton. Fitting with the principles of unobtrusive JavaScript, we’ll start with a semantically marked up list of restaurants, then use JavaScript to add the map, look up the restaurant locations and plot them as markers.
We’ll be using a couple of powerful tools. The first is jQuery, a JavaScript library that is ideally suited for unobtrusive scripting. jQuery allows us to manipulate elements on the page based on their CSS selector, which makes it easy to extract information from microformats.
The second is Mapstraction, introduced here by Andrew Turner a few days ago. We’ll be using Google Maps in the background, but Mapstraction makes it easy to change to a different provider if we want to later.
Getting Started
We’ll start off with a simple collection of microformatted restaurant details, representing my seven favourite restaurants in Brighton. The full, unstyled list can be seen in restaurants-plain.html. Each restaurant listing looks like this:
Since we’re dealing with a list of restaurants, each hCard is marked up inside a list item. Each restaurant is an organisation; we signify this by placing the classes fn and org on the element surrounding the restaurant’s name (according to the hCard spec, setting both fn and org to the same value signifies that the hCard represents an organisation rather than a person).
The address information itself is contained within a div of class adr. Note that the HTML element is not suitable here for two reasons: firstly, it is intended to mark up contact details for the current document rather than generic addresses; secondly, address is an inline element and as such cannot contain the paragraphs elements used here for the address information.
A nice thing about microformats is that they provide us with automatic hooks for our styling. For the moment we’ll just tidy up the whitespace a bit; for more advanced style tips consult John Allsop’s guide from 24 ways 2006.
.vcard p {
margin: 0;
}
.adr {
margin-bottom: 0.5em;
}
To plot the restaurants on a map we’ll need latitude and longitude for each one. We can find this out from their address using geocoding. Most mapping APIs include support for geocoding, which means we can pass the API an address and get back a latitude/longitude point. Mapstraction provides an abstraction layer around these APIs which can be included using the following script tag:
While we’re at it, let’s pull in the other external scripts we’ll be using:
That’s everything set up: let’s write some JavaScript!
In jQuery, almost every operation starts with a call to the jQuery function. The function simulates method overloading to behave in different ways depending on the arguments passed to it. When writing unobtrusive JavaScript it’s important to set up code to execute when the page has loaded to the point that the DOM is available to be manipulated. To do this with jQuery, pass a callback function to the jQuery function itself:
jQuery(function() {
// This code will be executed when the DOM is ready
});
Initialising the map
The first thing we need to do is initialise our map. Mapstraction needs a div with an explicit width, height and ID to show it where to put the map. Our document doesn’t currently include this markup, but we can insert it with a single line of jQuery code:
jQuery(function() {
// First create a div to host the map
var themap = jQuery('').css({
'width': '90%',
'height': '400px'
}).insertBefore('ul.restaurants');
});
While this is technically just a single line of JavaScript (with line-breaks added for readability) it’s actually doing quite a lot of work. Let’s break it down in to steps:
var themap = jQuery('')
Here’s jQuery’s method overloading in action: if you pass it a string that starts with a < it assumes that you wish to create a new HTML element. This provides us with a handy shortcut for the more verbose DOM equivalent:
var themap = document.createElement('div');
themap.id = 'themap';
Next we want to apply some CSS rules to the element. jQuery supports chaining, which means we can continue to call methods on the object returned by jQuery or any of its methods:
var themap = jQuery('').css({
'width': '90%',
'height': '400px'
})
Finally, we need to insert our new HTML element in to the page. jQuery provides a number of methods for element insertion, but in this case we want to position it directly before the
we are using to contain our restaurants. jQuery’s insertBefore() method takes a CSS selector indicating an element already on the page and places the current jQuery selection directly before that element in the DOM.
var themap = jQuery('').css({
'width': '90%',
'height': '400px'
}).insertBefore('ul.restaurants');
Finally, we need to initialise the map itself using Mapstraction. The Mapstraction constructor takes two arguments: the first is the ID of the element used to position the map; the second is the mapping provider to use (in this case google ):
// Initialise the map
var mapstraction = new Mapstraction('themap','google');
We want the map to appear centred on Brighton, so we’ll need to know the correct co-ordinates. We can use www.getlatlon.com to find both the co-ordinates and the initial map zoom level.
// Show map centred on Brighton
mapstraction.setCenterAndZoom(
new LatLonPoint(50.82423734980143, -0.14007568359375),
15 // Zoom level appropriate for Brighton city centre
);
We also want controls on the map to allow the user to zoom in and out and toggle between map and satellite view.
mapstraction.addControls({
zoom: 'large',
map_type: true
});
Adding the markers
It’s finally time to parse some microformats. Since we’re using hCard, the information we want is wrapped in elements with the class vcard. We can use jQuery’s CSS selector support to find them:
var vcards = jQuery('.vcard');
Now that we’ve found them, we need to create a marker for each one in turn. Rather than using a regular JavaScript for loop, we can instead use jQuery’s each() method to execute a function against each of the hCards.
jQuery('.vcard').each(function() {
// Do something with the hCard
});
Within the callback function, this is set to the current DOM element (in our case, the list item). If we want to call the magic jQuery methods on it we’ll need to wrap it in another call to jQuery:
jQuery('.vcard').each(function() {
var hcard = jQuery(this);
});
The Google maps geocoder seems to work best if you pass it the street address and a postcode. We can extract these using CSS selectors: this time, we’ll use jQuery’s find() method which searches within the current jQuery selection:
var streetaddress = hcard.find('.street-address').text();
var postcode = hcard.find('.postal-code').text();
The text() method extracts the text contents of the selected node, minus any HTML markup.
We’ve got the address; now we need to geocode it. Mapstraction’s geocoding API requires us to first construct a MapstractionGeocoder, then use the geocode() method to pass it an address. Here’s the code outline:
var geocoder = new MapstractionGeocoder(onComplete, 'google');
geocoder.geocode({'address': 'the address goes here');
The onComplete function is executed when the geocoding operation has been completed, and will be passed an object with the resulting point on the map. We just want to create a marker for the point:
var geocoder = new MapstractionGeocoder(function(result) {
var marker = new Marker(result.point);
mapstraction.addMarker(marker);
}, 'google');
For our purposes, joining the street address and postcode with a comma to create the address should suffice:
geocoder.geocode({'address': streetaddress + ', ' + postcode});
There’s one last step: when the marker is clicked, we want to display details of the restaurant. We can do this with an info bubble, which can be configured by passing in a string of HTML. We’ll construct that HTML using jQuery’s html() method on our hcard object, which extracts the HTML contained within that DOM node as a string.
var marker = new Marker(result.point);
marker.setInfoBubble(
'
' + hcard.html() + '
'
);
mapstraction.addMarker(marker);
We’ve wrapped the bubble in a div with class bubble to make it easier to style. Google Maps can behave strangely if you don’t provide an explicit width for your info bubbles, so we’ll add that to our CSS now:
.bubble {
width: 300px;
}
That’s everything we need: let’s combine our code together:
jQuery(function() {
// First create a div to host the map
var themap = jQuery('').css({
'width': '90%',
'height': '400px'
}).insertBefore('ul.restaurants');
// Now initialise the map
var mapstraction = new Mapstraction('themap','google');
mapstraction.addControls({
zoom: 'large',
map_type: true
});
// Show map centred on Brighton
mapstraction.setCenterAndZoom(
new LatLonPoint(50.82423734980143, -0.14007568359375),
15 // Zoom level appropriate for Brighton city centre
);
// Geocode each hcard and add a marker
jQuery('.vcard').each(function() {
var hcard = jQuery(this);
var streetaddress = hcard.find('.street-address').text();
var postcode = hcard.find('.postal-code').text();
var geocoder = new MapstractionGeocoder(function(result) {
var marker = new Marker(result.point);
marker.setInfoBubble(
'
' + hcard.html() + '
'
);
mapstraction.addMarker(marker);
}, 'google');
geocoder.geocode({'address': streetaddress + ', ' + postcode});
});
});
Here’s the finished code.
There’s one last shortcut we can add: jQuery provides the $ symbol as an alias for jQuery. We could just go through our code and replace every call to jQuery() with a call to $(), but this would cause incompatibilities if we ever attempted to use our script on a page that also includes the Prototype library. A more robust approach is to start our code with the following:
jQuery(function($) {
// Within this function, $ now refers to jQuery
// ...
});
jQuery cleverly passes itself as the first argument to any function registered to the DOM ready event, which means we can assign a local $ variable shortcut without affecting the $ symbol in the global scope. This makes it easy to use jQuery with other libraries.
Limitations of Geocoding
You may have noticed a discrepancy creep in to the last example: whereas my original list included seven restaurants, the geocoding example only shows five. This is because the Google Maps geocoder incorporates a rate limit: more than five lookups in a second and it starts returning error messages instead of regular results.
In addition to this problem, geocoding itself is an inexact science: while UK postcodes generally get you down to the correct street, figuring out the exact point on the street from the provided address usually isn’t too accurate (although Google do a pretty good job).
Finally, there’s the performance overhead. We’re making five geocoding requests to Google for every page served, even though the restaurants themselves aren’t likely to change location any time soon. Surely there’s a better way of doing this?
Microformats to the rescue (again)! The geo microformat suggests simple classes for including latitude and longitude information in a page. We can add specific points for each restaurant using the following markup:
E-Kagen
22-23 Sydney Street
Brighton, UK
BN1 4EN
Telephone: +44 (0)1273 687 068
Lat/Lon:
50.827917,
-0.137764
As before, I used www.getlatlon.com to find the exact locations – I find satellite view is particularly useful for locating individual buildings.
Latitudes and longitudes are great for machines but not so useful for human beings. We could hide them entirely with display: none, but I prefer to merely de-emphasise them (someone might want them for their GPS unit):
.vcard .geo {
margin-top: 0.5em;
font-size: 0.85em;
color: #ccc;
}
It’s probably a good idea to hide them completely when they’re displayed inside an info bubble:
.bubble .geo {
display: none;
}
We can extract the co-ordinates in the same way we extracted the address. Since we’re no longer geocoding anything our code becomes a lot simpler:
$('.vcard').each(function() {
var hcard = $(this);
var latitude = hcard.find('.geo .latitude').text();
var longitude = hcard.find('.geo .longitude').text();
var marker = new Marker(new LatLonPoint(latitude, longitude));
marker.setInfoBubble(
'
' + hcard.html() + '
'
);
mapstraction.addMarker(marker);
});
And here’s the finished geo example.
Further reading
We’ve only scratched the surface of what’s possible with microformats, jQuery (or just regular JavaScript) and a bit of imagination. If this example has piqued your interest, the following links should give you some more food for thought.
The hCard specification
Notes on parsing hCards
jQuery for JavaScript programmers – my extended tutorial on jQuery.
Dann Webb’s Sumo – a full JavaScript library for parsing microformats, based around some clever metaprogramming techniques.
Jeremy Keith’s Adactio Austin – the first place I saw using microformats to unobtrusively plot locations on a map. Makes clever use of hEvent as well.",2007,Simon Willison,simonwillison,2007-12-12T00:00:00+00:00,https://24ways.org/2007/unobtrusively-mapping-microformats-with-jquery/,code
249,Fast Autocomplete Search for Your Website,"Every website deserves a great search engine - but building a search engine can be a lot of work, and hosting it can quickly get expensive.
I’m going to build a search engine for 24 ways that’s fast enough to support autocomplete (a.k.a. typeahead) search queries and can be hosted for free. I’ll be using wget, Python, SQLite, Jupyter, sqlite-utils and my open source Datasette tool to build the API backend, and a few dozen lines of modern vanilla JavaScript to build the interface.
Try it out here, then read on to see how I built it.
First step: crawling the data
The first step in building a search engine is to grab a copy of the data that you plan to make searchable.
There are plenty of potential ways to do this: you might be able to pull it directly from a database, or extract it using an API. If you don’t have access to the raw data, you can imitate Google and write a crawler to extract the data that you need.
I’m going to do exactly that against 24 ways: I’ll build a simple crawler using wget, a command-line tool that features a powerful “recursive” mode that’s ideal for scraping websites.
We’ll start at the https://24ways.org/archives/ page, which links to an archived index for every year that 24 ways has been running.
Then we’ll tell wget to recursively crawl the website, using the --recursive flag.
We don’t want to fetch every single page on the site - we’re only interested in the actual articles. Luckily, 24 ways has nicely designed URLs, so we can tell wget that we only care about pages that start with one of the years it has been running, using the -I argument like this: -I /2005,/2006,/2007,/2008,/2009,/2010,/2011,/2012,/2013,/2014,/2015,/2016,/2017
We want to be polite, so let’s wait for 2 seconds between each request rather than hammering the site as fast as we can: --wait 2
The first time I ran this, I accidentally downloaded the comments pages as well. We don’t want those, so let’s exclude them from the crawl using -X ""/*/*/comments"".
Finally, it’s useful to be able to run the command multiple times without downloading pages that we have already fetched. We can use the --no-clobber option for this.
Tie all of those options together and we get this command:
wget --recursive --wait 2 --no-clobber
-I /2005,/2006,/2007,/2008,/2009,/2010,/2011,/2012,/2013,/2014,/2015,/2016,/2017
-X ""/*/*/comments""
https://24ways.org/archives/
If you leave this running for a few minutes, you’ll end up with a folder structure something like this:
$ find 24ways.org
24ways.org
24ways.org/2013
24ways.org/2013/why-bother-with-accessibility
24ways.org/2013/why-bother-with-accessibility/index.html
24ways.org/2013/levelling-up
24ways.org/2013/levelling-up/index.html
24ways.org/2013/project-hubs
24ways.org/2013/project-hubs/index.html
24ways.org/2013/credits-and-recognition
24ways.org/2013/credits-and-recognition/index.html
...
As a quick sanity check, let’s count the number of HTML pages we have retrieved:
$ find 24ways.org | grep index.html | wc -l
328
There’s one last step! We got everything up to 2017, but we need to fetch the articles for 2018 (so far) as well. They aren’t linked in the /archives/ yet so we need to point our crawler at the site’s front page instead:
wget --recursive --wait 2 --no-clobber
-I /2018
-X ""/*/*/comments""
https://24ways.org/
Thanks to --no-clobber, this is safe to run every day in December to pick up any new content.
We now have a folder on our computer containing an HTML file for every article that has ever been published on the site! Let’s use them to build ourselves a search index.
Building a search index using SQLite
There are many tools out there that can be used to build a search engine. You can use an open-source search server like Elasticsearch or Solr, a hosted option like Algolia or Amazon CloudSearch or you can tap into the built-in search features of relational databases like MySQL or PostgreSQL.
I’m going to use something that’s less commonly used for web applications but makes for a powerful and extremely inexpensive alternative: SQLite.
SQLite is the world’s most widely deployed database, even though many people have never even heard of it. That’s because it’s designed to be used as an embedded database: it’s commonly used by native mobile applications and even runs as part of the default set of apps on the Apple Watch!
SQLite has one major limitation: unlike databases like MySQL and PostgreSQL, it isn’t really designed to handle large numbers of concurrent writes. For this reason, most people avoid it for building web applications.
This doesn’t matter nearly so much if you are building a search engine for infrequently updated content - say one for a site that only publishes new content on 24 days every year.
It turns out SQLite has very powerful full-text search functionality built into the core database - the FTS5 extension.
I’ve been doing a lot of work with SQLite recently, and as part of that, I’ve been building a Python utility library to make building new SQLite databases as easy as possible, called sqlite-utils. It’s designed to be used within a Jupyter notebook - an enormously productive way of interacting with Python code that’s similar to the Observable notebooks Natalie described on 24 ways yesterday.
If you haven’t used Jupyter before, here’s the fastest way to get up and running with it - assuming you have Python 3 installed on your machine. We can use a Python virtual environment to ensure the software we are installing doesn’t clash with any other installed packages:
$ python3 -m venv ./jupyter-venv
$ ./jupyter-venv/bin/pip install jupyter
# ... lots of installer output
# Now lets install some extra packages we will need later
$ ./jupyter-venv/bin/pip install beautifulsoup4 sqlite-utils html5lib
# And start the notebook web application
$ ./jupyter-venv/bin/jupyter-notebook
# This will open your browser to Jupyter at http://localhost:8888/
You should now be in the Jupyter web application. Click New -> Python 3 to start a new notebook.
A neat thing about Jupyter notebooks is that if you publish them to GitHub (either in a regular repository or as a Gist), it will render them as HTML. This makes them a very powerful way to share annotated code. I’ve published the notebook I used to build the search index on my GitHub account.
Here’s the Python code I used to scrape the relevant data from the downloaded HTML files. Check out the notebook for a line-by-line explanation of what’s going on.
from pathlib import Path
from bs4 import BeautifulSoup as Soup
base = Path(""/Users/simonw/Dropbox/Development/24ways-search"")
articles = list(base.glob(""*/*/*/*.html""))
# articles is now a list of paths that look like this:
# PosixPath('...24ways-search/24ways.org/2013/why-bother-with-accessibility/index.html')
docs = []
for path in articles:
year = str(path.relative_to(base)).split(""/"")[1]
url = 'https://' + str(path.relative_to(base).parent) + '/'
soup = Soup(path.open().read(), ""html5lib"")
author = soup.select_one("".c-continue"")[""title""].split(
""More information about""
)[1].strip()
author_slug = soup.select_one("".c-continue"")[""href""].split(
""/authors/""
)[1].split(""/"")[0]
published = soup.select_one("".c-meta time"")[""datetime""]
contents = soup.select_one("".e-content"").text.strip()
title = soup.find(""title"").text.split("" ◆"")[0]
try:
topic = soup.select_one(
'.c-meta a[href^=""/topics/""]'
)[""href""].split(""/topics/"")[1].split(""/"")[0]
except TypeError:
topic = None
docs.append({
""title"": title,
""contents"": contents,
""year"": year,
""author"": author,
""author_slug"": author_slug,
""published"": published,
""url"": url,
""topic"": topic,
})
After running this code, I have a list of Python dictionaries representing each of the documents that I want to add to the index. The list looks something like this:
[
{
""title"": ""Why Bother with Accessibility?"",
""contents"": ""Web accessibility (known in other fields as inclus..."",
""year"": ""2013"",
""author"": ""Laura Kalbag"",
""author_slug"": ""laurakalbag"",
""published"": ""2013-12-10T00:00:00+00:00"",
""url"": ""https://24ways.org/2013/why-bother-with-accessibility/"",
""topic"": ""design""
},
{
""title"": ""Levelling Up"",
""contents"": ""Hello, 24 ways. Iu2019m Ashley and I sell property ins..."",
""year"": ""2013"",
""author"": ""Ashley Baxter"",
""author_slug"": ""ashleybaxter"",
""published"": ""2013-12-06T00:00:00+00:00"",
""url"": ""https://24ways.org/2013/levelling-up/"",
""topic"": ""business""
},
...
My sqlite-utils library has the ability to take a list of objects like this and automatically create a SQLite database table with the right schema to store the data. Here’s how to do that using this list of dictionaries.
import sqlite_utils
db = sqlite_utils.Database(""/tmp/24ways.db"")
db[""articles""].insert_all(docs)
That’s all there is to it! The library will create a new database and add a table to it called articles with the necessary columns, then insert all of the documents into that table.
(I put the database in /tmp/ for the moment - you can move it to a more sensible location later on.)
You can inspect the table using the sqlite3 command-line utility (which comes with OS X) like this:
$ sqlite3 /tmp/24ways.db
sqlite> .headers on
sqlite> .mode column
sqlite> select title, author, year from articles;
title author year
------------------------------ ------------ ----------
Why Bother with Accessibility? Laura Kalbag 2013
Levelling Up Ashley Baxte 2013
Project Hubs: A Home Base for Brad Frost 2013
Credits and Recognition Geri Coady 2013
Managing a Mind Christopher 2013
Run Ragged Mark Boulton 2013
Get Started With GitHub Pages Anna Debenha 2013
Coding Towards Accessibility Charlie Perr 2013
...
There’s one last step to take in our notebook. We know we want to use SQLite’s full-text search feature, and sqlite-utils has a simple convenience method for enabling it for a specified set of columns in a table. We want to be able to search by the title, author and contents fields, so we call the enable_fts() method like this:
db[""articles""].enable_fts([""title"", ""author"", ""contents""])
Introducing Datasette
Datasette is the open-source tool I’ve been building that makes it easy to both explore SQLite databases and publish them to the internet.
We’ve been exploring our new SQLite database using the sqlite3 command-line tool. Wouldn’t it be nice if we could use a more human-friendly interface for that?
If you don’t want to install Datasette right now, you can visit https://search-24ways.herokuapp.com/ to try it out against the 24 ways search index data. I’ll show you how to deploy Datasette to Heroku like this later in the article.
If you want to install Datasette locally, you can reuse the virtual environment we created to play with Jupyter:
./jupyter-venv/bin/pip install datasette
This will install Datasette in the ./jupyter-venv/bin/ folder. You can also install it system-wide using regular pip install datasette.
Now you can run Datasette against the 24ways.db file we created earlier like so:
./jupyter-venv/bin/datasette /tmp/24ways.db
This will start a local webserver running. Visit http://localhost:8001/ to start interacting with the Datasette web application.
If you want to try out Datasette without creating your own 24ways.db file you can download the one I created directly from https://search-24ways.herokuapp.com/24ways-ae60295.db
Publishing the database to the internet
One of the goals of the Datasette project is to make deploying data-backed APIs to the internet as easy as possible. Datasette has a built-in command for this, datasette publish. If you have an account with Heroku or Zeit Now, you can deploy a database to the internet with a single command. Here’s how I deployed https://search-24ways.herokuapp.com/ (running on Heroku’s free tier) using datasette publish:
$ ./jupyter-venv/bin/datasette publish heroku /tmp/24ways.db --name search-24ways
-----> Python app detected
-----> Installing requirements with pip
-----> Running post-compile hook
-----> Discovering process types
Procfile declares types -> web
-----> Compressing...
Done: 47.1M
-----> Launching...
Released v8
https://search-24ways.herokuapp.com/ deployed to Heroku
If you try this out, you’ll need to pick a different --name, since I’ve already taken search-24ways.
You can run this command as many times as you like to deploy updated versions of the underlying database.
Searching and faceting
Datasette can detect tables with SQLite full-text search configured, and will add a search box directly to the page. Take a look at http://search-24ways.herokuapp.com/24ways-b607e21/articles to see this in action.
SQLite search supports wildcards, so if you want autocomplete-style search where you don’t need to enter full words to start getting results you can add a * to the end of your search term. Here’s a search for access* which returns articles on accessibility:
http://search-24ways.herokuapp.com/24ways-ae60295/articles?_search=acces%2A
A neat feature of Datasette is the ability to calculate facets against your data. Here’s a page showing search results for svg with facet counts calculated against both the year and the topic columns:
http://search-24ways.herokuapp.com/24ways-ae60295/articles?_search=svg&_facet=year&_facet=topic
Every page visible via Datasette has a corresponding JSON API, which can be accessed using the JSON link on the page - or by adding a .json extension to the URL:
http://search-24ways.herokuapp.com/24ways-ae60295/articles.json?_search=acces%2A
Better search using custom SQL
The search results we get back from ../articles?_search=svg are OK, but the order they are returned in is not ideal - they’re actually being returned in the order they were inserted into the database! You can see why this is happening by clicking the View and edit SQL link on that search results page.
This exposes the underlying SQL query, which looks like this:
select rowid, * from articles where rowid in (
select rowid from articles_fts where articles_fts match :search
) order by rowid limit 101
We can do better than this by constructing a custom SQL query. Here’s the query we will use instead:
select
snippet(articles_fts, -1, 'b4de2a49c8', '8c94a2ed4b', '...', 100) as snippet,
articles_fts.rank, articles.title, articles.url, articles.author, articles.year
from articles
join articles_fts on articles.rowid = articles_fts.rowid
where articles_fts match :search || ""*""
order by rank limit 10;
You can try this query out directly - since Datasette opens the underling SQLite database in read-only mode and enforces a one second time limit on queries, it’s safe to allow users to provide arbitrary SQL select queries for Datasette to execute.
There’s a lot going on here! Let’s break the SQL down line-by-line:
select
snippet(articles_fts, -1, 'b4de2a49c8', '8c94a2ed4b', '...', 100) as snippet,
We’re using snippet(), a built-in SQLite function, to generate a snippet highlighting the words that matched the query. We use two unique strings that I made up to mark the beginning and end of each match - you’ll see why in the JavaScript later on.
articles_fts.rank, articles.title, articles.url, articles.author, articles.year
These are the other fields we need back - most of them are from the articles table but we retrieve the rank (representing the strength of the search match) from the magical articles_fts table.
from articles
join articles_fts on articles.rowid = articles_fts.rowid
articles is the table containing our data. articles_fts is a magic SQLite virtual table which implements full-text search - we need to join against it to be able to query it.
where articles_fts match :search || ""*""
order by rank limit 10;
:search || ""*"" takes the ?search= argument from the page querystring and adds a * to the end of it, giving us the wildcard search that we want for autocomplete. We then match that against the articles_fts table using the match operator. Finally, we order by rank so that the best matching results are returned at the top - and limit to the first 10 results.
How do we turn this into an API? As before, the secret is to add the .json extension. Datasette actually supports multiple shapes of JSON - we’re going to use ?_shape=array to get back a plain array of objects:
JSON API call to search for articles matching SVG
The HTML version of that page shows the time taken to execute the SQL in the footer. Hitting refresh a few times, I get response times between 2 and 5ms - easily fast enough to power a responsive autocomplete feature.
A simple JavaScript autocomplete search interface
I considered building this using React or Svelte or another of the myriad of JavaScript framework options available today, but then I remembered that vanilla JavaScript in 2018 is a very productive environment all on its own.
We need a few small utility functions: first, a classic debounce function adapted from this one by David Walsh:
function debounce(func, wait, immediate) {
let timeout;
return function() {
let context = this, args = arguments;
let later = () => {
timeout = null;
if (!immediate) func.apply(context, args);
};
let callNow = immediate && !timeout;
clearTimeout(timeout);
timeout = setTimeout(later, wait);
if (callNow) func.apply(context, args);
};
};
We’ll use this to only send fetch() requests a maximum of once every 100ms while the user is typing.
Since we’re rendering data that might include HTML tags (24 ways is a site about web development after all), we need an HTML escaping function. I’m amazed that browsers still don’t bundle a default one of these:
const htmlEscape = (s) => s.replace(
/>/g, '>'
).replace(
/Autocomplete search
And now the autocomplete implementation itself, as a glorious, messy stream-of-consciousness of JavaScript:
// Embed the SQL query in a multi-line backtick string:
const sql = `select
snippet(articles_fts, -1, 'b4de2a49c8', '8c94a2ed4b', '...', 100) as snippet,
articles_fts.rank, articles.title, articles.url, articles.author, articles.year
from articles
join articles_fts on articles.rowid = articles_fts.rowid
where articles_fts match :search || ""*""
order by rank limit 10`;
// Grab a reference to the
const searchbox = document.getElementById(""searchbox"");
// Used to avoid race-conditions:
let requestInFlight = null;
searchbox.onkeyup = debounce(() => {
const q = searchbox.value;
// Construct the API URL, using encodeURIComponent() for the parameters
const url = (
""https://search-24ways.herokuapp.com/24ways-866073b.json?sql="" +
encodeURIComponent(sql) +
`&search=${encodeURIComponent(q)}&_shape=array`
);
// Unique object used just for race-condition comparison
let currentRequest = {};
requestInFlight = currentRequest;
fetch(url).then(r => r.json()).then(d => {
if (requestInFlight !== currentRequest) {
// Avoid race conditions where a slow request returns
// after a faster one.
return;
}
let results = d.map(r => `
`).join("""");
document.getElementById(""results"").innerHTML = results;
});
}, 100); // debounce every 100ms
There’s just one more utility function, used to help construct the HTML results:
const highlight = (s) => htmlEscape(s).replace(
/b4de2a49c8/g, ''
).replace(
/8c94a2ed4b/g, ''
);
This is what those unique strings passed to the snippet() function were for.
Avoiding race conditions in autocomplete
One trick in this code that you may not have seen before is the way race-conditions are handled. Any time you build an autocomplete feature, you have to consider the following case:
User types acces
Browser sends request A - querying documents matching acces*
User continues to type accessibility
Browser sends request B - querying documents matching accessibility*
Request B returns. It was fast, because there are fewer documents matching the full term
The results interface updates with the documents from request B, matching accessibility*
Request A returns results (this was the slower of the two requests)
The results interface updates with the documents from request A - results matching access*
This is a terrible user experience: the user saw their desired results for a brief second, and then had them snatched away and replaced with those results from earlier on.
Thankfully there’s an easy way to avoid this. I set up a variable in the outer scope called requestInFlight, initially set to null.
Any time I start a new fetch() request, I create a new currentRequest = {} object and assign it to the outer requestInFlight as well.
When the fetch() completes, I use requestInFlight !== currentRequest to sanity check that the currentRequest object is strictly identical to the one that was in flight. If a new request has been triggered since we started the current request we can detect that and avoid updating the results.
It’s not a lot of code, really
And that’s the whole thing! The code is pretty ugly, but when the entire implementation clocks in at fewer than 70 lines of JavaScript, I honestly don’t think it matters. You’re welcome to refactor it as much you like.
How good is this search implementation? I’ve been building search engines for a long time using a wide variety of technologies and I’m happy to report that using SQLite in this way is genuinely a really solid option. It scales happily up to hundreds of MBs (or even GBs) of data, and the fact that it’s based on SQL makes it easy and flexible to work with.
A surprisingly large number of desktop and mobile applications you use every day implement their search feature on top of SQLite.
More importantly though, I hope that this demonstrates that using Datasette for an API means you can build relatively sophisticated API-backed applications with very little backend programming effort. If you’re working with a small-to-medium amount of data that changes infrequently, you may not need a more expensive database. Datasette-powered applications easily fit within the free tier of both Heroku and Zeit Now.
For more of my writing on Datasette, check out the datasette tag on my blog. And if you do build something fun with it, please let me know on Twitter.",2018,Simon Willison,simonwillison,2018-12-19T00:00:00+00:00,https://24ways.org/2018/fast-autocomplete-search-for-your-website/,code
326,Don't be eval(),"JavaScript is an interpreted language, and like so many of its peers it includes the all powerful eval() function. eval() takes a string and executes it as if it were regular JavaScript code. It’s incredibly powerful and incredibly easy to abuse in ways that make your code slower and harder to maintain. As a general rule, if you’re using eval() there’s probably something wrong with your design.
Common mistakes
Here’s the classic misuse of eval(). You have a JavaScript object, foo, and you want to access a property on it – but you don’t know the name of the property until runtime. Here’s how NOT to do it:
var property = 'bar';
var value = eval('foo.' + property);
Yes it will work, but every time that piece of code runs JavaScript will have to kick back in to interpreter mode, slowing down your app. It’s also dirt ugly.
Here’s the right way of doing the above:
var property = 'bar';
var value = foo[property];
In JavaScript, square brackets act as an alternative to lookups using a dot. The only difference is that square bracket syntax expects a string.
Security issues
In any programming language you should be extremely cautious of executing code from an untrusted source. The same is true for JavaScript – you should be extremely cautious of running eval() against any code that may have been tampered with – for example, strings taken from the page query string. Executing untrusted code can leave you vulnerable to cross-site scripting attacks.
What’s it good for?
Some programmers say that eval() is B.A.D. – Broken As Designed – and should be removed from the language. However, there are some places in which it can dramatically simplify your code. A great example is for use with XMLHttpRequest, a component of the set of tools more popularly known as Ajax. XMLHttpRequest lets you make a call back to the server from JavaScript without refreshing the whole page. A simple way of using this is to have the server return JavaScript code which is then passed to eval(). Here is a simple function for doing exactly that – it takes the URL to some JavaScript code (or a server-side script that produces JavaScript) and loads and executes that code using XMLHttpRequest and eval().
function evalRequest(url) {
var xmlhttp = new XMLHttpRequest();
xmlhttp.onreadystatechange = function() {
if (xmlhttp.readyState==4 && xmlhttp.status==200) {
eval(xmlhttp.responseText);
}
}
xmlhttp.open(""GET"", url, true);
xmlhttp.send(null);
}
If you want this to work with Internet Explorer you’ll need to include this compatibility patch.",2005,Simon Willison,simonwillison,2005-12-07T00:00:00+00:00,https://24ways.org/2005/dont-be-eval/,code
129,Knockout Type - Thin Is Always In,"OS X has gorgeous native anti-aliasing (although I will admit to missing 10px aliased Geneva — *sigh*). This is especially true for dark text on a light background. However, things can go awry when you start using light text on a dark background. Strokes thicken. Counters constrict. Letterforms fill out like seasonal snackers.
So how do we combat the fat? In Safari and other Webkit-based browsers we can use the CSS ‘text-shadow’ property. While trying to add a touch more contrast to the navigation on haveamint.com I noticed an interesting side-effect on the weight of the type.
The second line in the example image above has the following style applied to it:
This creates an invisible drop-shadow. (Why is it invisible? The shadow is positioned directly behind the type (the first two zeros) and has no spread (the third zero). So the color, black, is completely eclipsed by the type it is supposed to be shadowing.)
Why applying an invisible drop-shadow effectively lightens the weight of the type is unclear. What is clear is that our light-on-dark text is now of a comparable weight to its dark-on-light counterpart.
You can see this trick in effect all over ShaunInman.com and in the navigation on haveamint.com and Subtraction.com. The HTML and CSS source code used to create the example images used in this article can be found here.",2006,Shaun Inman,shauninman,2006-12-17T00:00:00+00:00,https://24ways.org/2006/knockout-type/,code
316,Have Your DOM and Script It Too,"When working with the XMLHttpRequest object it appears you can only go one of three ways:
You can stay true to the colorful moniker du jour and stick strictly to the responseXML property
You can play with proprietary – yet widely supported – fire and inject the value of responseText property into the innerHTML of an element of your choosing
Or you can be eval() and parse JSON or arbitrary JavaScript delivered via responseText
But did you know that there’s a fourth option giving you the best of the latter two worlds? Mint uses this unmentioned approach to grab fresh HTML and run arbitrary JavaScript simultaneously. Without relying on eval(). “But wait-”, you might say, “when would I need to do this?” Besides the example below this technique is handy for things like tab groups that need initialization onload but miss the main onload event handler by a mile thanks to asynchronous scripting.
Consider the problem
Originally Mint used option 2 to refresh or load new tabs into individual Pepper panes without requiring a full roundtrip to the server. This was all well and good until I introduced the new Client Mode which when enabled allows anyone to view a Mint installation without being logged in. If voyeurs are afoot as Client Mode is disabled, the next time they refresh a pane the entire login page is inserted into the current document. That’s not very helpful so I needed a way to redirect the current document to the login page.
Enter the solution
Wouldn’t it be cool if browsers interpreted the contents of script tags crammed into innerHTML? Sure, but unfortunately, that just wasn’t meant to be. However like the body element, image elements have an onload event handler. When the image has fully loaded the handler runs the code applied to it. See where I’m going with this?
By tacking a tiny image (think single pixel, transparent spacer gif – shudder) onto the end of the HTML returned by our Ajax call, we can smuggle our arbitrary JavaScript into the existing document. The image is added to the DOM, and our stowaway can go to town.
This is the results of our Ajax call.
Please be neat
So we’ve just jammed some meaningless cruft into our DOM. If our script does anything with images this addition could have some unexpected side effects. (Remember The Fly?) So in order to save that poor, unsuspecting element whose innerHTML we just swapped out from sharing Jeff Goldblum’s terrible fate we should tidy up after ourselves. And by using the removeChild method we do just that.
This is the results of our Ajax call.
",2005,Shaun Inman,shauninman,2005-12-24T00:00:00+00:00,https://24ways.org/2005/have-your-dom-and-script-it-too/,code
298,First Steps in VR,"The web is all around us. As web folk, it is our responsibility to consider the impact our work can have. Part of this includes thinking about the future; the web changes lives and if we are building the web then we are the ones making decisions that affect people in every corner of the world. I find myself often torn between wanting to make the right decisions, and just wanting to have fun. To fiddle and play. We all know how important it is to sometimes just try ideas, whether they will amount to much or not.
I think of these two mindsets as production and prototyping, though of course there are lots of overlap and phases in between. I mention this because virtual reality is currently seen as a toy for rich people, and in some ways at the moment it is. But with WebVR we are able to create interesting experiences with a relatively low entry point. I want us to have open minds, play around with things, and then see how we can use the tools we have at our disposal to make things that will help people.
Every year we see articles saying it will be the “year of virtual reality”, that was especially prevalent this year. 2016 has been a year of progress, VR isn’t quite mainstream but with efforts like Playstation VR and Google Cardboard, we are definitely seeing much more of it. This year also saw the consumer editions of the Oculus Rift and HTC Vive. So it does seem to be a good time for an overview of how to get involved with creating virtual reality on the web.
WebVR is an API for connecting to devices and retrieving continuous data such as the position and orientation. Unlike the Web Audio API and some other APIs, WebVR does not feel like a framework. You use it however you want, taking the data and using it as you wish. To make it easier, there are plenty of resources such as Three.js, A-Frame and ReactVR that help to make the heavy lifting a bit easier.
Getting Started with A-Frame
I like taking the opportunity to learn new things whenever I can. So while planning this article I thought that instead of trying to teach WebGL or even Three.js in a way that is approachable for all, I would create my first project using A-Frame and write about that. This is not a tutorial as such, I just want to show how to go about getting involved with VR. The beauty of A-Frame is that it is very similar to web components, you can just write HTML to build worlds that will automatically work on all the different types of devices. It uses WebGL and WebVR but in such a way that it quite drastically reduces the learning curve. That’s not to say you can’t build complex things, you have complete access to write JavaScript and shaders.
I’m lazy. Whenever I learn a new language or framework I have found that the best way, personally, for me to learn is to have a project and to copy the starting code from someone else. A project lets you have a good idea of what you want to produce and it means you can ignore a lot of the irrelevant documentation, focussing purely on what you need. That reduces the stress of figuring things out. Copying code also makes it easier, because you know your boilerplate code is working. There’s nothing worse than getting stuck before anything actually works the first time. So I tinker. I take code and I modify it, I play around. It’s fun.
For this project I wanted to keep things as simple as possible, so I can easily explain it without the classic “draw a circle then draw an owl”. I wrote a list of requirements, with some stretch goals that you can give a try yourself if you fancy:
Must work on Google Cardboard at a minimum, because of price
Therefore, it must not rely on having a controller
Auto-moving around a maze would be a good example
Move in direction you look
Stretch goal: Scoring, time until you hit a wall or get stuck in maze
Stretch goal: Levels, so the map doesn’t need to be random
Stretch goal: Snow!
I decided to base this project on an example, Platforms, by Don McCurdy who wrote the really useful aframe-extras. Platforms has random 3D blocks that you can jump onto, going up into the sky. So I took his code and reduced it so that the blocks are randomly spread on the ground.
24 ways
As you can see, this is very readable. Especially if you ignore the JavaScript that is used to create the maze. A-Frame (with A-Frame Extras) gives you a lot of power with relatively little to learn. We start with an which is the container for everything that is going to show up on the screen. There are a few which can be compared to
as they are essentially non-semantic containers, able to be used for any purpose. The attributes are used to define functionality, for example the camera attribute sets the entity to function as a camera and kinematic-body makes it collide instead of go through objects. Attributes are also used to set position and sizes, often using JavaScript to dynamically define them.
Styling
Now we’ve got the HTML written, we need to style it. To do this we add A-Frame compatible attributes such as color and material. I recommend playing around, you can get some quite impressive effects fairly easily. Originally I wanted a light snowy maze but it ended up being dark and foggy, as I really liked the feeling it gave.
Note, you will probably need a server running for images to work. You can do this by running python -m ""SimpleHTTPServer"" in the folder where the code is, then go to localhost:8000 in browser.
Textures
Unless you are going for a cartoony style, you probably want to find some textures. I found some on textures.com, one image worked well for the walls and the other for the floor.
The is used to define (as well as preload and cache) all assets, including images, audio and video. As you can see, images in the Asset Management System just use normal img tags. The ids are important here as we can use them later for using the textures.
To apply a texture to an object, you create a material. For a simple material where it just shows the image, you set the src to the id selector of the image.
Replace:
With:
This will automatically make the image repeat over the entire floor, in my case filling it with bricks. The walls are pretty much identical, with the slight exception that it is set in JavaScript as they are dynamically defined.
box.setAttribute('material', 'src: #texture-wall');
That’s it for the textures, for now at least. These will not look completely realistic, as the light will bump off the rectangular wall rather than texture itself. This can be improved by using maps, textures that are used to modify the shape and physical properties of the object.
Lighting
The next part of styling is lighting. By using fog and different types of lighting, we are able to add atmospheric details to the game to make it feel that bit more realistic and polished.
There are lots of types of light in A-Frame (most coming from Three.js). You can add a light either by using the entity or by attaching a light attribute to any other entity. If there are no lights defined then A-Frame adds some by default so that the scene is always lit.
To start with I wanted to light up the scene with a general light, type=""ambient"", so that the whole game felt slightly dark. I chose to set the light to a reddish colour #92455E. After playing around with intensity I chose 0.4, it added enough light to get the feeling I wanted without it being overly red. I also added a blue skybox (), as it looked a bit odd with a white sky.
I felt that the maze looked good with a red tinge but it was a bit flat, everything was the same colour and it was a bit dark. So I added a light within the #player entity, this could have been as an attribute but I set it as a child a-light instead. By using type=""point"" with a high intensity and low distance, it showed close walls as being lighter. It also added a sort-of object to the player, it isn’t a walking human or anything but by moving light where the player is it feels a bit more physical.
By this point it was starting to look decent, so I wanted to add the fog to really give some personality and depth to the maze. To do this I added the fog attribute to the with type=exponential so it looks thicker the further away it is and a mid intensity, so you feel a bit lost but can still see.
I was very happy with this result. It took a lot of playing around with colours and values, which is fun in itself. I highly recommend you take the code (or write your own) and play around with the numbers.
Movement
One of the reasons I decided to use aframe-extras is that it has a few different camera controls built in. As you saw earlier, I am using the universal-controls which gives WASD (keyboard) controls by default. I wanted to make it automatically move in the direction that you’re looking, but I wasn’t quite sure how without rewriting the controls. So I asked Don McCurdy for advice and he very nicely gave me a small snippet of code to get it working.
AFRAME.registerComponent('automove-controls', {
init: function () {
this.speed = 0.1;
this.isMoving = true;
this.velocityDelta = new THREE.Vector3();
},
isVelocityActive: function () {
return this.isMoving;
},
getVelocityDelta: function () {
this.velocityDelta.z = this.isMoving ? -speed : 0;
return this.velocityDelta.clone();
}
});
Replace:
universal-controls
With:
universal-controls=""movementControls: automove, gamepad, keyboard""
This works by creating a component automove-controls that adds auto-move to the player without overriding movement completely. It doesn’t even touch direction, it just checks if isMoving is true then moves the player by the set speed. Components can be creating for adding all kinds of functionality with relative ease. It makes it very powerful for people of all difficulty levels.
Building a map
Currently the maze is created randomly, which is great but means there will often be walls that overlap or the player gets trapped with nowhere to go. So to solve this, I decided to use a map editor (Tiled) so that we can create the mazes ourselves. This is a great start towards one of the stretch goals, levels.
I made the maze in Tiled by finding a random tileset online (we don’t need to actually show the images), I used one tile for the wall and another for the player. Then I exported as a JavaScript file and modified it in my text editor to get rid of everything I didn’t need. I made it so 0 is the path, 1 is the wall and 2 is the player. I then added the script to the HTML, as a separate file so it’s easy to update in the future.
var map =
{
""data"":[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
""height"":10,
""width"":10
}
As you can see, this gives a simple 10x10 maze with some dead ends. The player starts in the bottom right corner (my choice, could be anywhere). I rewrote the random platforms code (from Don’s example) to instead loop over the map data and place walls where it is 1 and position the player where data is 2. I set the position so that the origin of the map would be 0,1.5,0. The y axis is in this case the height (ground being 0), but if a wall is positioned at 0 by its centre then some of it is underground. So the y needed to be the height divided by 2.
document.querySelector('a-scene').addEventListener('render-target-loaded', function () {
var WALL_SIZE = 5,
WALL_HEIGHT = 3;
var el = document.querySelector('#walls');
var wall;
for (var x = 0; x < map.height; x++) {
for (var y = 0; y < map.width; y++) {
var i = y*map.width + x;
var position = (x-map.width/2)*WALL_SIZE + ' ' + 1.5 + ' ' + (y-map.height/2)*WALL_SIZE;
if (map.data[i] === 1) {
// Create wall
wall = document.createElement('a-box');
el.appendChild(wall);
wall.setAttribute('color', '#fff');
wall.setAttribute('material', 'src: #texture-wall;');
wall.setAttribute('width', WALL_SIZE);
wall.setAttribute('height', WALL_HEIGHT);
wall.setAttribute('depth', WALL_SIZE);
wall.setAttribute('position', position);
wall.setAttribute('static-body', ');
}
if (map.data[i] === 2) {
// Set player position
document.querySelector('#player').setAttribute('position', position);
}
}
}
console.info('Walls added.');
});
With this added, it makes it nice and easy to change around the map as well as to add new features. Perhaps you want monsters or objects. Just set the number in the map data and add an if statement to the loop. In the future you could add layers, so multiple things can be in the same position. Or perhaps even make the maze go up the y axis too, with ramps or staircases. There’s a lot you can do with relative ease. As you can see, A-Frame really does reduce the learning curve of 3D and VR on the web.
It’s Not All Fun And Games
A lot of examples of virtual reality are games, including this one. So it is understandable to think that VR is for gaming, but actually that’s just a tiny subset. There are all sorts of applications for VR, including story telling, data visualisation and even meditation.
There have been a number of cases where it has been shown virtual reality can help as a tool for therapies:
Oxford study finds virtual reality can help treat severe paranoia
Virtual Reality Therapy for Phobias at the Duke Faculty Practice
Bravemind: Virtual Reality Exposure Therapy at the University of Southern California
These are just a few examples of where virtual reality is being used around the world to help people feel better and get through some very tough times. There have also been examples of it being used for simulating war zones or medical situations, both as a teaching and journalism tool.
Wrapping Up
Ten years ago, on this very site, Cameron Moll wrote an article explaining the mobile web. He explained how mobile phones with data plans were becoming increasingly common, that WAP 2.0 included the XHTML Mobile Profile meaning it would be familiar with web folk. “The mobile web is rapidly becoming an XHTML environment, and thus you and I can apply our existing “desktop web” skills to understand how to develop content for it.”
We can look at that and laugh a little, we have come a very long way in the last decade. Even people in developing countries with very little money have mobile phones with access to a web that is far more capable than the “desktop web” Cameron was referring to.
So while I am not saying virtual reality is going to change the world or replace our phones, who knows! We can use our skills as web folk to dabble, we don’t need to learn any new languages. If on the 2026 edition of 24 ways, somebody references this article and looks at how far we have come… well, let’s hope we have used our skills well and made the world just that little bit better. And if VR is a fad? Well it’s fun… have a go anyway.",2016,Shane Hudson,shanehudson,2016-12-11T00:00:00+00:00,https://24ways.org/2016/first-steps-in-vr/,code
211,Automating Your Accessibility Tests,"Accessibility is one of those things we all wish we were better at. It can lead to a bunch of questions like: how do we make our site better? How do we test what we have done? Should we spend time each day going through our site to check everything by hand? Or just hope that everyone on our team has remembered to check their changes are accessible?
This is where automated accessibility tests can come in. We can set up automated tests and have them run whenever someone makes a pull request, and even alongside end-to-end tests, too.
Automated tests can’t cover everything however; only 20 to 50% of accessibility issues can be detected automatically. For example, we can’t yet automate the comparison of an alt attribute with an image’s content, and there are some screen reader tests that need to be carried out by hand too. To ensure our site is as accessible as possible, we will still need to carry out manual tests, and I will cover these later.
First, I’m going to explain how I implemented automated accessibility tests on Elsevier’s ecommerce pages, and share some of the lessons I learnt along the way.
Picking the right tool
One of the hardest, but most important parts of creating our automated accessibility tests was choosing the right tool.
We began by investigating aXe CLI, but soon realised it wouldn’t fit our requirements. It couldn’t check pages that required a visitor to log in, so while we could test our product pages, we couldn’t test any customer account pages. Instead we moved over to Pa11y. Its beforeScript step meant we could log into the site and test pages such as the order history.
The example below shows the how the beforeScript step completes a login form and then waits for the login to complete before testing the page:
beforeScript: function(page, options, next) {
// An example function that can be used to make sure changes have been confirmed before continuing to run Pa11y
function waitUntil(condition, retries, waitOver) {
page.evaluate(condition, function(err, result) {
if (result || retries < 1) {
// Once the changes have taken place continue with Pa11y testing
waitOver();
} else {
retries -= 1;
setTimeout(function() {
waitUntil(condition, retries, waitOver);
}, 200);
}
});
}
// The script to manipulate the page must be run with page.evaluate to be run within the context of the page
page.evaluate(function() {
const user = document.querySelector('#login-form input[name=""email""]');
const password = document.querySelector('#login-form input[name=""password""]');
const submit = document.querySelector('#login-form input[name=""submit""]');
user.value = 'user@example.com';
password.value = 'password';
submit.click();
}, function() {
// Use the waitUntil function to set the condition, number of retries and the callback
waitUntil(function() {
return window.location.href === 'https://example.com';
}, 20, next);
});
}
The waitUntil callback allows the test to be delayed until our test user is successfully logged in.
Another thing to consider when picking a tool is the type of error messages it produces. aXe groups all elements with the same error together, so the list of issues is a lot easier to read, and it’s easier to identify the most commons problems. For example, here are some elements that have insufficient colour contrast:
Violation of ""color-contrast"" with 8 occurrences!
Ensures the contrast between foreground and background colors meets
WCAG 2 AA contrast ratio thresholds. Correct invalid elements at:
- #maincontent > .make_your_mark > div:nth-child(2) > p > span > span
- #maincontent > .make_your_mark > div:nth-child(4) > p > span > span
- #maincontent > .inform_your_decisions > div:nth-child(2) > p > span > span
- #maincontent > .inform_your_decisions > div:nth-child(4) > p > span > span
- #maincontent > .inform_your_decisions > div:nth-child(6) > p > span > span
- #maincontent > .inform_your_decisions > div:nth-child(8) > p > span > span
- #maincontent > .inform_your_decisions > div:nth-child(10) > p > span > span
- #maincontent > .inform_your_decisions > div:nth-child(12) > p > span > span
For details, see: https://dequeuniversity.com/rules/axe/2.5/color-contrast
aXe also provides links to their site where they discuss the best way to fix the problem.
In comparison, Pa11y lists each individual error which can lead to a very verbose list. However, it does provide helpful suggestions of how to fix problems, such as suggesting an alternative shade of a colour to use:
• Error: This element has insufficient contrast at this conformance level.
Expected a contrast ratio of at least 4.5:1, but text in this element has a contrast ratio of 2.96:1.
Recommendation: change text colour to #767676.
⎣ WCAG2AA.Principle1.Guideline1_4.1_4_3.G18.Fail
⎣ #maincontent > div:nth-child(10) > div:nth-child(8) > p > span > span
⎣ Featured products:
Integrating into our build pipeline
We decided the perfect time to run our accessibility tests would be alongside our end-to-end tests. We have a Jenkins job that detects changes to our staging site and then triggers the end-to-end tests, and in turn our accessibility tests. Our Jenkins job retrieves the contents of a GitHub repository containing our Pa11y script file and npm package manifest.
Once Jenkins has cloned the repository, it installs any dependencies and executes the tests via:
npm install && npm test
Bundling the URLs to be tested into our test script means we don’t have a command line style test where we list each URL we wish to test in the Jenkins CLI. It’s an effective method but can also be cluttered, and obscure which URLs are being tested.
In the middle of the office we have a monitor displaying a Jenkins dashboard and from this we can see if the accessibility tests are passing or failing. Everyone in the team has access to the Jenkins logs and when the build fails they can see why and fix the issue.
Fixing the issues
As mentioned earlier, Pa11y can generate a long list of areas for improvement which can be very verbose and quite overwhelming. I recommend going through the list to see which issues occur most frequently and fix those first. For example, we initially had a lot of errors around colour contrast, and one shade of grey in particular. By making this colour darker, the number of errors decreased, and we could focus on the remaining issues.
Another thing I like to do is to tackle the quick fixes, such as adding alt text to images. These are small things that allow us to make an impact instantly, giving us time to fix more detailed concerns such as addressing tabindex issues, or speaking to our designers about changing the contrast of elements on the site.
Manual testing
Adding automated tests to check our site for accessibility is great, but as I mentioned earlier, this can only cover 20-50% of potential issues. To improve on this, we need to test by hand too, either by ourselves or by asking others.
One way we can test our site is to throw our mouse or trackpad away and interact with the site using only a keyboard. This allows us to check items such as tab order, and ensure menu items, buttons etc. can be used without a mouse. The commands may be different on different operating systems, but there are some great guides online for learning more about these.
It’s tempting to add alt text and aria-labels to make errors go away, but if they don’t make any sense, what use are they really? Using a screenreader we can check that alt text accurately represents the image. This is also a great way to double check that our ARIA roles make sense, and that they correctly identify elements and how to interact with them. When testing our site with screen readers, it’s important to remember that not all screen readers are the same and some may interact with our site differently to others.
Consider asking a range of people with different needs and abilities to test your site, too. People experience the web in numerous ways, be they permanent, temporary or even situational. They may interact with your site in ways you hadn’t even thought about, so this is a good way to broaden your knowledge and awareness.
Tips and tricks
One of our main issues with Pa11y is that it may find issues we don’t have the power to solve. A perfect example of this is the one pixel image Facebook injects into our site. So, we wrote a small function to go though such errors and ignore the ones that we cannot fix.
const test = pa11y({
....
hideElements: '#ratings, #js-bigsearch',
...
});
const ignoreErrors: string[] = [
'',
'',
''
];
const filterResult = result => {
if (ignoreErrors.indexOf(result.context) > -1) {
return false;
}
return true;
};
Initially we wanted to focus on fixing the major problems, so we added a rule to ignore notices and warnings. This made the list or errors much smaller and allowed us focus on fixing major issues such as colour contrast and missing alt text. The ignored notices and warnings can be added in later after these larger issues have been resolved.
const test = pa11y({
ignore: [
'notice',
'warning'
],
...
});
Jenkins gotchas
While using Jenkins we encountered a few problems. Sometimes Jenkins would indicate a build had passed when in reality it had failed. This was because Pa11y had timed out due to PhantomJS throwing an error, or the test didn’t go past the first URL. Pa11y has recently released a new beta version that uses headless Chrome instead of PhantomJS, so hopefully these issues will less occur less often.
We tried a few approaches to solve these issues. First we added error handling, iterating over the array of test URLs so that if an unexpected error happened, we could catch it and exit the process with an error indicating that the job had failed (using process.exit(1)).
for (const url of urls) {
try {
console.log(url);
let urlResult = await run(url);
urlResult = urlResult.filter(filterResult);
urlResult.forEach(result => console.log(result));
}
catch (e) {
console.log('Error:', e);
process.exit(1);
}
}
We also had issues with unhandled rejections sometimes caused by a session disconnecting or similar errors. To avoid Jenkins indicating our site was passing with 100% accessibility, when in reality it had not executed any tests, we instructed Jenkins to fail the job when an unhandled rejection or uncaught exception occurred:
process.on('unhandledRejection', (reason, p) => {
console.log('Unhandled Rejection at:', p, 'reason:', reason);
process.exit(1);
});
process.on('uncaughtException', (err) => {
console.log('Caught exception: ${err}n');
process.exit(1);
});
Now it’s your turn
That’s it! That’s how we automated accessibility testing for Elsevier ecommerce pages, allowing us to improve our site and make it more accessible for everyone. I hope our experience can help you automate accessibility tests on your own site, and bring the web a step closer to being accessible to all.",2017,Seren Davies,serendavies,2017-12-07T00:00:00+00:00,https://24ways.org/2017/automating-your-accessibility-tests/,code
295,Internet of Stranger Things,"This year I’ve been running a workshop about using JavaScript and Node.js to work with all different kinds of electronics on the Raspberry Pi. So especially for 24 ways I’m going to show you how I made a very special Raspberry Pi based internet connected project! And nothing says Christmas quite like a set of fairy lights connected to another dimension1.
What you’ll see
You can rig up the fairy lights in your home, with the scrawly letters written under each one. The people from the other side (i.e. the internet) will be able to write messages to you from their browser in real time. In fact why not try it now; check this web page. When you click the lights in your browser, my lights (and yours) will turn on and off in real life! (There may be a queue if there are lots of people accessing it, hit the “Send a message” button and wait your turn.)
It’s all done with JavaScript, using Node.js running on both the Raspberry Pi and on the server. I’m using WebSockets to communicate in real time between the browser, server and Raspberry Pi.
What you’ll need
Raspberry Pi any of the following models: Zero (will need straight male header pins soldered2 and Micro USB OTG adaptor), A+, B+, 2, or 3
Micro SD card at least 4Gb Class 10 speed3
Micro USB power supply at least 2A
USB Wifi dongle (unless you have a Pi 3 - that has wifi built in).
Addressable fairy lights
Logic level shifter (with pins soldered unless you want to do it!)
Breadboard
Jumper wires (3x male to male and 4x female to male)
Optional but recommended
Base board to hold the Pi and Breadboard (often comes with a breadboard!)
Find links for where to buy all of these items that goes along with this tutorial. The total price should be around $1004.
Setting up the Raspberry Pi
You’ll need to install the SD card for the Raspberry Pi. You’ll find a link to download a disk image on the support document, ready-made with the Raspbian version of Linux, along with Node.js and all the files you need. Download it and write it to the SD card using the fantastic free software Etcher5.
Next up you have to configure the wifi details on the SD card. If you plug the card into your computer you should see a drive called BOOT. There’s a text file on there called wpa_supplicant.conf. Open it up in your favourite text editor and replace mywifi and mypassword with your wifi details6.
network={
ssid=""mywifi""
psk=""mypassword""
}
Save the file, eject the card from your computer and plug it into the Raspberry Pi.
If you have a base board or holder for the Raspberry Pi, attach it now. Then connect the wifi USB dongle7 and power supply, but don’t plug it in yet!
Wiring!
Time to wire everything up!
First of all, push the Logic Level Converter into the middle of the breadboard:
Logic Level Converter
The logic level converter may be labelled differently from the one in the diagram but the pins are usually exactly the same internally. I would just make sure the pins marked HV (High Voltage) are on the bottom and LV (Low Voltage) are on the top.
Raspberry Pi pins only output 3.3v but the lights need 5v. That’s why we need the logic level converter in there to boost up the signal.
Connect the first two wires between the Raspberry Pi pins and the breadboard:
Note that the pins on the Raspberry Pi are male, so you need a female to male jumper wire to connect between them and the breadboard. The colours don’t have to match but it’s easier to follow (and check) if you use the same ones as in the diagram.
Then the next two:
This is what you should have so far:
Lights
Now to connect the lights! My ones have a connector with three holes in it that I can push jumper wires into, and hopefully yours will too! So I used the male-to-male jumper wires to connect them to the breadboard.
Make sure that you connect the right end of the lights, mine has a male connector at the wrong end so it’s impossible to do this, but double check.
Also make sure that the holes in the light connector are the same as mine. To do this, follow the wires from the connector to the first light and look at the circuit board inside. You should just about be able to make out the connections labelled + (sometimes 5V, V+ or VCC), GND (or ‘-’ or G) and DI (sometimes DIN for data in).
You can just about make out the +, DI and GND on this picture. Note that on the other side of the board there is a DO for data out - that’s what takes the data along to the chip in the next light. Make sure that you’re plugging into the data-in and not the data-out!
That’s it! Everything’s plugged in and ready to go! But before you plug power into your Pi, double check all your wires and make sure they’re exactly right! You could damage your Raspberry Pi if it is not wired correctly. So triple check!
The Moment of Truth!
Plug in the Raspberry Pi and wait around a minute or two for it to boot up. If all is well, the lights should strobe rainbow colours for one second - that’s your confirmation that it’s connected to my WebSocket server and ready to receive messages from the upside-down!
However, if the first light in the string is pulsing red, it means that you’re not connected to the internet. So check the Troubleshooting section of the support document. If it’s pulsing green then you’re connected to the internet but can’t connect to my server. It must have gone down. Sorry! The code will keep trying so leave it running and maybe it’ll come back up.
Rig up the lights!
Fix the lights up on the wall however you want, pins, nails, tape. I’ve used cable clips. Just be careful! I’m using a 50 light string so I’ve programmed it to use the lights at the end for the letters. That way I have just under half the string to extend down to the floor where I can keep the Raspberry Pi.
Check the photo here to see how the lights line up, note that there are spare unused lights in-between each row:
Now visit lights.seb.ly and you’ll see this :
If you’re the only one online you’ll have direct connection to the lights and any letter you click on will light up both in the browser and in real life. If there are other people there, you’ll need to click the button to join the queue and wait your turn.
How it works - the geeky details!
Electronics:
The pins on the Raspberry Pi are known as GPIO pins, general-purpose input/output. You can connect a wide variety of electronic components to them, LED lights, buttons, switches, and sensors. You can turn the power to the pins on and off using Node.js (or Python, if you prefer).
Addressable LEDs or “Neopixels”
We’re only using one GPIO pin on the Raspberry Pi (the other connections are 5V, 3.3V and ground) and that single pin is controlling all of the lights in the string. The code turns the pin on and off really fast in strictly timed morse-code-like dots and dashes to transmit binary data. The chips attached to each LED decode the binary and adjust the output to the LED accordingly. That chip then sends the data on to the next light in the string.
The chips on each light are the WS2811, part of the WS281x family that come in a multitude of different form factors and are often packaged with tiny LEDs in a single component. They are commonly referred to as Neopixels8 and I used them on my Laser Light Synths project.
Neopixels with the chip and the LED all in one - it’s the white square shaped component and the darker square inside is the chip. These are only 5mm wide!
A Laser Light Synth! Covered with around 800 super bright neopixels!
Logic Level Converter
The logic level converter is a really cheap and easy way to change the level from 3.3v to 5v and back again. You must be careful that you do not connect 5v into a GPIO pin or you will most likely damage the Raspberry Pi processor chip.
Power
Neopixels can often draw a lot of current so you need to be careful how you power them. I’ve measured the current draw from the string to be less than 800mA so you should be fine wired directly to the 5V output. But if you use more lights or have them all on really bright at once, you’ll need to use a separate 5V power supply. If you want to learn more, check out Adafruit’s Neopixel Uberguide.
Node.js
There are two Node.js apps running here, one on the Raspberry Pi and one on my server. You can see the code on my GitHub at github.com/sebleedelisle/stranger-lights for the Raspberry Pi and github.com/sebleedelisle/stranger-lights-server for the server. And they’re hosted on npm as stranger-lights and stranger-lights-server.
The server side code sets up a standard web server to deliver the HTML for the web interface. It also sets up a WebSocket server that allows for real-time communication between the browser and the server. This server code also manages the queue and who is in control of the lights at any given time.
WebSockets
I’m using the excellent Socket.io library to manage the WebSocket connection. Both the browser and the Raspberry Pi Node.js app connects to my WebSocket server.
When you click on a letter in the browser, a message is sent to the server, which forwards it to the connected Raspberry Pi clients and also all the web browsers9.
The Raspberry Pi code
The Node.js app runs automatically on startup, and I made this happen by adding this to the /etc/rc.local file:
node /home/pi/strangerthings/client.js > /dev/null &
Anything in the rc.local file gets executed when the Pi boots up and this line of code runs the Node.js app and routes its output to nowhere (ie /dev/null). The & means that it runs it in the background and doesn’t hold up the boot process.
Working with the Raspberry Pi headless
You might know that when a computer has no screen or keyboard, you would refer to it as “running headless”. So just like most web servers, you need to configure it over the network with ssh10. If you’re on a mac you can find your Pi on the network through the name raspberrypi.local11, otherwise you’ll need to find its IP address. There’s more on the guide to Remote Access instructions on the Raspberry Pi website. And if you’re very new to the terminal, I highly recommend this great online Linux command line tutorial.
Improvements
This is quite an early experiment and I’m sure I’ll discover lots of optimisations over the next few weeks, especially if the server gets a proper hammering today! But there are a few things you can do. Obviously I’ve just rigged up my lights with Post-it notes. It’d be a lot nicer to get a paint brush and try to recreate the Winona-in-a-manic-state text style.
Where next?
Finding quality resources about Node.js for electronics on the Pi can be somewhat hit and miss, but this is getting better all the time. Alternatively I am thinking about running some online courses, please let me know if that’s something you’d be interested in, or sign up to my mailing list at st4i.com.
There are many many more resources for the Raspberry Pi with Python (gpiozero is a good place to start), so if that language works for you, you’ll be spoilt for choice!
Also take a look at Arduino - it’s an incredibly popular platform for electronics and the internet is literally bursting with resources.
I hope you enjoyed this little foray into the world of JavaScript electronics on the Raspberry Pi! If you get this working at home please let me know! Tweet me at @seb_ly.
Not a particularly original idea, but I don’t think I’ve seen anyone do it quite like this before, ie using WebSockets, and Node.js on a Raspberry Pi. Other examples: Internet of Stranger Things, Strangerlights.com, and loads of examples on Instructables ↩︎
Video guide to soldering pins on to a Pi Zero and further soldering advice from Adafruit ↩︎
Slower cards will work but performance may suffer ↩︎
Or £5,000 in UK money. Sorry, Brexit joke :) ↩︎
You will need a card reader on your computer - most micro SD cards come with an adaptor that fits standard SD slots. ↩︎
SSID and password should be all that you need but you can see all the config options on this wpa supplicant guide ↩︎
Raspberry Pi Zero will require the OTG to USB adaptor to attach the wifi dongle ↩︎
Thanks to Adafruit who invented the term neopixels so we don’t have to refer to them as WS281x any more! ↩︎
So you can see other people sending messages in the browser ↩︎
ssh is short for Secure Shell and is a way to connect to a remote computer and type in it just like you would in the terminal. ↩︎
You can change this default hostname using raspi-config ↩︎",2016,Seb Lee-Delisle,sebleedelisle,2016-12-01T00:00:00+00:00,https://24ways.org/2016/internet-of-stranger-things/,code
221,"“Probably, Maybe, No”: The State of HTML5 Audio","With the hype around HTML5 and CSS3 exceeding levels not seen since 2005’s Ajax era, it’s worth noting that the excitement comes with good reason: the two specifications render many years of feature hacks redundant by replacing them with native features. For fun, consider how many CSS2-based rounded corners hacks you’ve probably glossed over, looking for a magic solution. These days, with CSS3, the magic is border-radius (and perhaps some vendor prefixes) followed by a coffee break.
CSS3’s border-radius, box-shadow, text-shadow and gradients, and HTML5’s