| 20 |
Make Your Browser Dance |
It was a crisp winter’s evening when I pulled up alongside the pier. I stepped out of my car and the bitterly cold sea air hit my face. I walked around to the boot, opened it and heaved out a heavy flight case. I slammed the boot shut, locked the car and started walking towards the venue.
This was it. My first gig. I thought about all those weeks of preparation: editing video clips, creating 3-D objects, making coloured patterns, then importing them all into software and configuring effects to change as the music did; targeting frequency, beat, velocity, modifying size, colour, starting point; creating playlists of these… and working out ways to mix them as the music played.
This was it. This was me VJing.
This was all a lifetime (well a decade!) ago.
When I started web designing, VJing took a back seat. I was more interested in interactive layouts, semantic accessible HTML, learning all the IE bugs and mastering the quirks that CSS has to offer. More recently, I have been excited by background gradients, 3-D transforms, the @keyframe directive, as well as new APIs such as getUserMedia, indexedDB, the Web Audio API
But wait, have I just come full circle? Could it be possible, with these wonderful new things in technologies I am already familiar with, that I could VJ again, right here, in a browser?
Well, there’s only one thing to do: let’s try it!
Let’s take to the dance floor
Over the past couple of years working in The Lab I have learned to take a much more iterative approach to projects than before. One of my new favourite methods of working is to create a proof of concept to make sure my theory is feasible, before going on to create a full-blown product. So let’s take the same approach here.
The main VJing functionality I want to recreate is manipulating visuals in relation to sound. So for my POC I need to create a visual, with parameters that can be changed, then get some sound and see if I can analyse that sound to detect some data, which I can then use to manipulate the visual parameters. Easy, right?
So, let’s start at the beginning: creating a simple visual. For this I’m going to create a CSS animation. It’s just a funky i element with the opacity being changed to make it flash.
See the Pen Creating a light by Rumyra (@Rumyra) on CodePen
A note about prefixes: I’ve left them out of the code examples in this post to make them easier to read. Please be aware that you may need them. I find a great resource to find out if you do is caniuse.com. You can also check out all the code for the examples in this article
Start the music
Well, that’s pretty easy so far. Next up: loading in some sound. For this we’ll use the Web Audio API. The Web Audio API is based around the concept of nodes. You have a source node: the sound you are loading in; a destination node: usually the device’s speakers; and any number of processing nodes in between. All this processing that goes on with the audio is sandboxed within the AudioContext.
So, let’s start by initialising our audio context.
var contextClass = window.AudioContext;
if (contextClass) {
//web audio api available.
var audioContext = new contextClass();
} else {
//web audio api unavailable
//warn user to upgrade/change browser
}
Now let’s load our sound file into the new context we created with an XMLHttpRequest.
function loadSound() {
//set audio file url
var audioFileUrl = '/octave.ogg';
//create new request
var request = new XMLHttpRequest();
request.open("GET", audioFileUrl, true);
request.responseType = "arraybuffer";
request.onload = function() {
//take from http request and decode into buffer
context.decodeAudioData(request.response, function(buffer) {
audioBuffer = buffer;
});
}
request.send();
}
Phew! Now we’ve loaded in some sound! There are plenty of things we can do with the Web Audio API: increase volume; add filters; spatialisation. If you want to dig deeper, the O’Reilly Web Audio API book by Boris Smus is available to read online free.
All we really want to do for this proof of concept, however, is analyse the sound data. To do this we really need to know what data we have.
Learning the steps
Let’s take a minute to step back and remember our school days and science class. I’m sure if I drew a picture of a sound wave, we would all start nodding our heads.
The sound you hear is caused by pressure differences in the particles in the air. Sound pushes these particles together, causing vibrations. Amplitude is basically strength of pressure. A simple example of change of amplitude is when you increase the volume on your stereo and the output wave increases in size.
This is great when everything is analogue, but the waveform varies continuously and it’s not suitable for digital processing: there’s an infinite set of values. For digital processing, we need discrete numbers.
We have to sample the waveform at set time intervals, and record data such as amplitude and frequency. Luckily for us, just the fact we have a digital sound file means all this hard work is done for us. What we’re doing in the code above is piping that data in the audio context. All we need to do now is access it.
We can do this with the Web Audio API’s analysing functionality. Just pop in an analysing node before we connect the source to its destination node.
function createAnalyser(source) {
//create analyser node
analyser = audioContext.createAnalyser();
//connect to source
source.connect(analyzer);
//pipe to speakers
analyser.connect(audioContext.destination);
}
The data I’m really interested in here is frequency. Later we could look into amplitude or time, but for now I’m going to stick with frequency.
The analyser node gives us frequency data via the getFrequencyByteData method.
Don’t forget to count!
To collect the data from the getFrequencyByteData method, we need to pass in an empty array (a JavaScript typed array is ideal). But how do we know how many items the array will need when we create it?
This is really up to us and how high the resolution of frequencies we want to analyse is. Remember we talked about sampling the waveform; this happens at a certain rate (sample rate) which you can find out via the audio context’s sampleRate attribute. This is good to bear in mind when you’re thinking about your resolution of frequencies.
var sampleRate = audioContext.sampleRate;
Let’s say your file sample rate is 48,000, making the maximum frequency in the file 24,000Hz (thanks to a wonderful theorem from Dr Harry Nyquist, the maximum frequency in the file is always half the sample rate). The analyser array we’re creating will contain frequencies up to this point. This is ideal as the human ear hears the range 0–20,000hz.
So, if we create an array which has 2,400 items, each frequency recorded will be 10Hz apart. However, we are going to create an array which is half the size of the FFT (fast Fourier transform), which in this case is 2,048 which is the default. You can set it via the fftSize property.
//set our FFT size
analyzer.fftSize = 2048;
//create an empty array with 1024 items
var frequencyData = new Uint8Array(1024);
So, with an array of 1,024 items, and a frequency range of 24,000Hz, we know each item is 24,000 ÷ 1,024 = 23.44Hz apart.
The thing is, we also want that array to be updated constantly. We could use the setInterval or setTimeout methods for this; however, I prefer the new and shiny requestAnimationFrame.
function update() {
//constantly getting feedback from data
requestAnimationFrame(update);
analyzer.getByteFrequencyData(frequencyData);
}
Putting it all together
Sweet sticks! Now we have an array of frequencies from the sound we loaded, updating as the sound plays. Now we want that data to trigger our animation from earlier.
We can easily pause and run our CSS animation from JavaScript:
element.style.webkitAnimationPlayState = "paused";
element.style.webkitAnimationPlayState = "running";
Unfortunately, this may not be ideal as our animation might be a whole heap longer than just a flashing light. We may want to target specific points within that animation to have it stop and start in a visually pleasing way and perhaps not smack bang in the middle.
There is no really easy way to do this at the moment as Zach Saucier explains in this wonderful article. It takes some jiggery pokery with setInterval to try to ascertain how far through the CSS animation you are in percentage terms.
This seems a bit much for our proof of concept, so let’s backtrack a little. We know by the animation we’ve created which CSS properties we want to change. This is pretty easy to do directly with JavaScript.
element.style.opacity = "1";
element.style.opacity = "0.2";
So let’s start putting it all together. For this example I want to trigger each light as a different frequency plays. For this, I’ll loop through the HTML elements and change the opacity style if the frequency gain goes over a certain threshold.
//get light elements
var lights = document.getElementsByTagName('i');
var totalLights = lights.length;
for (var i=0; i<totalLights; i++) {
//get frequencyData key
var freqDataKey = i*8;
//if gain is over threshold for that frequency animate light
if (frequencyData[freqDataKey] > 160){
//start animation on element
lights[i].style.opacity = "1";
} else {
lights[i].style.opacity = "0.2";
}
}
See all the code in action here. I suggest viewing in a modern browser :)
Awesome! It is true — we can VJ in our browser!
Let’s dance!
So, let’s start to expand this simple example. First, I feel the need to make lots of lights, rather than just a few. Also, maybe we should try a sound file more suited to gigs or clubs.
Check it out!
I don’t know about you, but I’m pretty excited — that’s just a bit of HTML, CSS and JavaScript!
The other thing to think about, of course, is the sound that you would get at a venue. We don’t want to load sound from a file, but rather pick up on what is playing in real time. The easiest way to do this, I’ve found, is to capture what my laptop’s mic is picking up and piping that back into the audio context. We can do this by using getUserMedia.
Let’s include this in this demo. If you make some noise while viewing the demo, the lights will start to flash.
And relax :)
There you have it. Sit back, play some music and enjoy the Winamp like experience in front of you.
So, where do we go from here? I already have a wealth of ideas. We haven’t started with canvas, SVG or the 3-D features of CSS. There are other things we can detect from the audio as well. And yes, OK, it’s questionable whether the browser is the best environment for this. For one, I’m using a whole bunch of nonsensical HTML elements (maybe each animation could be held within a web component in the future). But hey, it’s fun, and it looks cool and sometimes I think it’s OK to just dance. |
2013 |
Ruth John |
ruthjohn |
2013-12-02T00:00:00+00:00 |
https://24ways.org/2013/make-your-browser-dance/ |
code |
| 70 |
Bringing Your Code to the Streets |
— or How to Be a Street VJ
Our amazing world of web code is escaping out of the browser at an alarming rate and appearing in every aspect of the environment around us. Over the past few years we’ve already seen JavaScript used server-side, hardware coded with JavaScript, a rise of native style and desktop apps created with HTML, CSS and JavaScript, and even virtual reality (VR) is getting its fair share of front-end goodness.
You can go ahead and play with JavaScript-powered hardware such as the Tessel or the Espruino to name a couple. Just check out the Tessel project page to see JavaScript in the world of coffee roasting or sleep tracking your pet. With the rise of the internet of things, JavaScript can be seen collecting information on flooding among other things. And if that’s not enough ‘outside the browser’ implementations, Node.js servers can even be found in aircraft!
I previously mentioned VR and with three.js’s extra StereoEffect.js module it’s relatively simple to get browser 3D goodness to be Google Cardboard-ready, and thus set the stage for all things JavaScript and VR. It’s been pretty popular in the art world too, with interactive works such as Seb Lee-Delisle’s Lunar Trails installation, featuring the old arcade game Lunar Lander, which you can now play in your browser while others watch (it is the web after all). The Science Museum in London held Chrome Web Lab, an interactive exhibition featuring five experiments, showcasing the magic of the web. And it’s not even the connectivity of the web that’s being showcased; we can even take things offline and use web code for amazing things, such as fighting Ebola.
One thing is for sure, JavaScript is awesome. Hell, if you believe those telly programs (as we all do), JavaScript can even take down the stock market, purely through the witchcraft of canvas! Go JavaScript!
Now it’s our turn
So I wanted to create a little project influenced by this theme, and as it’s Christmas, take it to the streets for a little bit of party fun! Something that could take code anywhere. Here’s how I made a portable visual projection pack, a piece of video mixing software and created some web-coded street art.
Step one: The equipment
You will need:
One laptop: with HDMI output and a modern browser installed, such as Google Chrome.
One battery-powered mini projector: I’ve used a Texas Instruments DLP; for its 120 lumens it was the best cost-to-lumens ratio I could find.
One MIDI controller (optional): mine is an ICON iDJ as it suits mixing visuals. However, there is more affordable hardware on the market such as an Akai LPD8 or a Korg nanoPAD2. As you’ll see in the article, this is optional as it can be emulated within the software.
A case to carry it all around in.
Step two: The software
The projected visuals, I imagined, could be anything you can create within a browser, whether that be simple HTML and CSS, images, videos, SVG or canvas. The only requirement I have is that they move or change with sound and that I can mix any one visual into another.
You may remember a couple of years ago I created a demo on this very site, allowing audio-triggered visuals from the ambient sounds your device mic was picking up. That was a great starting point – I used that exact method to pick up the audio and thus the first requirement was complete. If you want to see some more examples of visuals I’ve put together for this, there’s a showcase on CodePen.
The second requirement took a little more thought. I needed two screens, which could at any point show any of the visuals I had coded, but could be mixed from one into the other and back again. So let’s start with two divs, both absolutely positioned so they’re on top of each other, but at the start the second screen’s opacity is set to zero.
Now all we need is a slider, which when moved from one side to the other slowly sets the second screen’s opacity to 1, thereby fading it in.
See the Pen Mixing Screens (Software Version) by Rumyra (@Rumyra) on CodePen.
Mixing Screens (CodePen)
As you saw above, I have a MIDI controller and although the software method works great, I’d quite like to make use of this nifty piece of kit. That’s easily done with the Web MIDI API. All I need to do is call it, and when I move one of the sliders on the controller (I’ve allocated the big cross fader in the middle for this), pick up on the change of value and use that to control the opacity instead.
var midi, data;
// start talking to MIDI controller
if (navigator.requestMIDIAccess) {
navigator.requestMIDIAccess({
sysex: false
}).then(onMIDISuccess, onMIDIFailure);
} else {
alert(“No MIDI support in your browser.”);
}
// on success
function onMIDISuccess(midiData) {
// this is all our MIDI data
midi = midiData;
var allInputs = midi.allInputs.values();
// loop over all available inputs and listen for any MIDI input
for (var input = allInputs.next(); input && !input.done; input = allInputs.next()) {
// when a MIDI value is received call the onMIDIMessage function
input.value.onmidimessage = onMIDIMessage;
}
}
function onMIDIMessage(message) {
// data comes in the form [command/channel, note, velocity]
data = message.data;
// Opacity change for screen. The cross fader values are [176, 8, {0-127}]
if ( (data[0] === 176) && (data[1] === 8) ) {
// this value will change as the fader is moved
var opacity = data[2]/127;
screenTwo.style.opacity = opacity;
}
}
The final code was slightly more complicated than this, as I decided to switch the two screens based on the frequencies of the sound that was playing, and use the cross fader to depict the frequency threshold value. This meant they flickered in and out of each other, rather than just faded. There’s a very rough-and-ready first version of the software on GitHub.
Phew, Great! Now we need to get all this to the streets!
Step three: Portable kit
Did you notice how I mentioned a case to carry it all around in? I wanted the case to be morphable, so I could use the equipment from it too, a sort of bag-to-usherette-tray-type affair. Well, I had an unused laptop bag…
I strengthened it with some MDF, so when I opened the bag it would hold like a tray where the laptop and MIDI controller would sit. The projector was Velcroed to the external pocket of the bag, so when it was a tray it would project from underneath. I added two durable straps, one for my shoulders and one round my waist, both attached to the bag itself. There was a lot of cutting and trimming. As it was a laptop bag it was pretty thick to start and sewing was tricky. However, I only broke one sewing machine needle; I’ve been known to break more working with leather, so I figured I was doing well. By the way, you can actually buy usherette trays, but I just couldn’t resist hacking my own :)
Step four: Take to the streets
First, make sure everything is charged – everything – a lot! The laptop has to power both the MIDI controller and the projector, and although I have a mobile phone battery booster pack, that’ll only charge the projector should it run out. I estimated I could get a good hour of visual artistry before I needed to worry, though.
I had a couple of ideas about time of day and location. Here in the UK at this time of year, it gets dark around half past four, so I could easily head out in a city around 5pm and it would be dark enough for the projections to be seen pretty well. I chose Bristol, around the waterfront, as there were some interesting locations to try it out in. The best was Millennium Square: busy but not crowded and plenty of surfaces to try projecting on to.
My first time out with the portable audio/visual pack (PAVP as it will now be named) was brilliant. I played music and projected visuals, like a one-woman band of A/V!
You might be thinking what the point of this was, besides, of course, it being a bit of fun. Well, this project got me to look at canvas and SVG more closely. The Web MIDI API was really interesting; MIDI as a data format has some great practical uses. I think without our side projects we may not have all these wonderful uses for our everyday code. Not only do they remind us coding can, and should, be fun, they also help us learn and grow as makers.
My favourite part? When I was projecting into a water feature in Millennium Square. For those who are familiar, you’ll know it’s like a wall of water so it produced a superb effect. I drew quite a crowd and a kid came to stand next to me and all I could hear him say with enthusiasm was, ‘Oh wow! That’s so cool!’
Yes… yes, kid, it was cool. Making things with code is cool.
Massive thanks to the lovely Drew McLellan for his incredibly well-directed photography, and also Simon Johnson who took a great hand in perfecting the kit while it was attached. |
2015 |
Ruth John |
ruthjohn |
2015-12-06T00:00:00+00:00 |
https://24ways.org/2015/bringing-your-code-to-the-streets/ |
code |
| 248 |
How to Use Audio on the Web |
I know what you’re thinking. I never never want to hear sound anywhere near a browser, ever ever, wow! 🙉
You’re having flashbacks, flashbacks to the days of yore, when we had a <bgsound> element and yup did everyone think that was the most rad thing since <blink>. I mean put those two together with a <marquee>, only use CSS colour names, make sure your borders were all set to ridge and you’ve got yourself the neatest website since 1998.
The sound played when the website loaded and you could play a MIDI file as well! Everyone could hear that wicked digital track you chose. Oh, surfing was gnarly back then.
Yes it is 2018, the end of in fact, soon to be 2019. We are certainly living in the future. Hoverboards self driving cars, holodecks VR headsets, rocket boots drone racing, sound on websites get real, Ruth.
We can’t help but be jaded, even though the <bgsound> element is depreciated, and the autoplay policy appeared this year. Although still in it’s infancy, the policy “controls when video and audio is allowed to autoplay”, which should reduce the somewhat obtrusive playing of sound when a website or app loads in the future.
But then of course comes the question, having lived in a muted present for so long, where and why would you use audio?
✨ Showcase Time ✨
There are some incredible uses of audio on websites today. This is my personal favourite futurelibrary.no, a site from Norway chronicling books that have been published from a forest of trees planted precisely for the books themselves. The sound effects are lovely, adding to the overall experience.
futurelibrary.no
Another site that executes this well is pottermore.com. The Hogwarts WebGL simulation uses both sound effects and ambient background music and gives a great experience. The button hovers are particularly good.
pottermore.com
Eighty-six and a half years is a beautiful narrative site, documenting the musings of an eighty-six and a half year old man. The background music playing on this site is not offensive, it adds to the experience.
Eighty-six and a half years
Sound can be powerful and in some cases useful. Last year I wrote about using them to help validate forms. Audiochart is a library which “allows the user to explore charts on web pages using sound and the keyboard”. Ben Byford recorded voice descriptions of the pages on his website for playback should you need or want it. There is a whole area of accessibility to be explored here.
Then there’s education. Fancy beginning with some piano in the new year? flowkey.com is a website which allows you to play along and learn at the same time. Need to brush up on your music theory? lightnote.co takes you through lessons to do just that, all audio enhanced. Electronic music more your thing? Ableton has your back with learningmusic.ableton.com, a site which takes you through the process of composing electronic music. A website, all made possible through the powers with have with the Web Audio API today.
lightnote.co
learningmusic.ableton.com
Considerations
Yes, tis the season, let’s be more thoughtful about our audios. There are some user experience patterns to begin with. 86andahalfyears.com tells the user they are about to ‘enter’ the site and headphones are recommended. This is a good approach because it a) deals with the autoplay policy (audio needs to be instigated by a user gesture) and b) by stating headphones are recommended you are setting the users expectations, they will expect sound, and if in a public setting can enlist the use of a common electronic device to cause less embarrassment.
Eighty-six and a half years
Allowing mute and/or volume control clearly within the user interface is a good idea. It won’t draw the user out of the experience, it’ll give more control to the user about what audio they want to hear (they may not want to turn down the volume of their entire device), and it’s less thought to reach for a very visible volume than to fumble with device settings.
Indicating that sound is playing is also something to consider. Browsers do this by adding icons to tabs, but this isn’t always the first place to look for everyone.
To The Future
So let’s go!
We see amazing demos built with Web Audio, and I’m sure, like me, they make you think, oh wow I wish I could do that / had thought of that / knew the first thing about audio to begin to even conceive that.
But audio doesn’t actually need to be all bells and whistles (hey, it’s Christmas). Starting, stopping and adjusting simple panning and volume might be all you need to get started to introduce some good sound design in your web design.
Isn’t it great then that there’s a tutorial just for that! Head on over to the MDN Web Audio API docs where the Using the Web Audio API article takes you through playing and pausing sounds, volume control and simple panning (moving the sound from left to right on stereo speakers).
This year I believe we have all experienced the web as a shopping mall more than ever. It’s shining store fronts, flashing adverts, fast food, loud noises.
Let’s use 2019 to create more forests to explore, oceans to dive and mountains to climb. |
2018 |
Ruth John |
ruthjohn |
2018-12-22T00:00:00+00:00 |
https://24ways.org/2018/how-to-use-audio-on-the-web/ |
design |