Sunday, October 25, 2009

Dynamically setting the Yaws log level

Debugging a web-based application is more often than not done by first analyzing log files. Even when you can attach a REPL to a running application. They are an integral part of the developer's tool set.

But production applications usually do not enable tracing by default, because this could degrade performance and waste essential resources (memory, CPU time, disk, etc.). Would you dump the content of every HTTP request received by a web server on disk? Of course, when problems occur, trace level is simply raised, with the hope that the logs will contain some useful debugging info.

With the Yaws Erlang-based web server, the log level can be set programmatically like this:
-module(my_yaws_utils).
-export([set_yaws_trace_level/1]).

set_yaws_trace_level(Level) ->
{ok, GC, Groups} = yaws_api:getconf(),
yaws_config:hard_setconf(GC#gconf{trace=Level}, Groups).

with Level being either none, {true, http}, or {true, traffic}. The first value disables tracing, the second enables tracing of the HTTP requests in file trace.http, while the last enables tracing of the whole traffic (requests and responses).

It is certainly possible to attach to the Yaws server from another Erlang node to call this function. But if your application is administered by someone not really used to Erlang, it may be more appropriate to provide a command-line script for that purpose.

Basically, the script would connect to the Yaws node (named yaws_daemon@lelouch) and invoke the function with the appropriate log level:

#!/bin/sh
node=yaws_daemon@lelouch
# we should check the number of arguments to the script here...
case $1 in
none) level=none;;
http) level="{true, http}";;
traffic) level="{true, traffic}"
esac

erl -sname trace_script -noinput \
-setcookie ErlangCookie \
-eval "rpc:call($node, my_yaws_utils, set_yaws_trace_level, [$level])." \
-s init stop

The call to rpc:call does the actual RPC call to set the log level on the Yaws server.

The cool thing is the fact that this technique can be applied to a whole bunch of other utility tools, like scripts to stop the server, to reload the application configuration, etc. And this applies to all Erlang applications as well, this is in no way specific to Yaws.

And there are probably other ways to do the same thing. Let me know if you have a better solution.

9 comments:

Alexandre Abreu said...

You probably meant:

[...] GC#gconf{trace=Level} [...]

Dominique Boucher said...

You're right! Fixed.

Thanks!

Steve Vinoski said...

Your description mentions a call to net_kernel:connect but the example script doesn't show it.

Dominique Boucher said...

@Steve Oups! Thanks for noticing. My first version of the script was making use of net_kernel:connect but that was useless. Fixed.

Alexandre Abreu said...

I am curious, are you using this setup (the Erlang + Yaws) in a production environment at work or is it a home project?

The core of my question is: if it is at work, how do you manage to introduce this kind of tools at work (tools that are foreign to most companies) without frightening the management or your peer workers?

I mean, I know a bit about your company (I work in the same building) and you guys are top devs but Erlang/Yaws seems like a far stretch from your typical J2EE setup,

Just curious,

Dominique Boucher said...

Hi Alexandre!

Both, actually. At home, I use Erlang+Yaws for the MSLUG website, as well as a number of other tools, like monitoring some web sites and notifying me via IM, VoiceXML, or mail...

At work, we use this setup for NuGram Hosted Server (http://www.grammarserver.com). The front-end is written in Erlang, while the grammar-related stuff is implemented in Java + Kawa Scheme. (Nice isn't it?)

I have been able to convince the management team to use Erlang for a number of reasons. First, Erlang was a natural fit for the problems we had (I could write a post just on this very subject). Also, most of my team members were willing to use a functional programming language, they all had some experience with Lisp/Prolog/Haskell/Python... It was an easy sell, and I needed their buy in before pushing the idea at the management level. Finally, I am part of the management team... that helped a bit, but I had to prepare a good case anyway.

But most importantly, the management team is very open to trying new things. It remains that the development of NuGram Hosted Server was seen more like an experiment. We don't sell this version of NuGram Server. The real product is still based on standard J2EE technologies. And the use of Scheme/Erlang is still limited to a single dev team, they have not spread across the entire organization.

Alexandre Abreu said...

Thanks for your answer,

"First, Erlang was a natural fit for the problems we had (I could write a post just on this very subject)"

That would be great!

Alexandre Abreu said...

Some additional questions (answer if you can / want off course):

- how do erlang and kawa scheme interop? http / json ? ffi ?

- to follow up on that, why erlang / yaws and not scala / lift to have a better integration with the jvm ?

- the server part if hosted on 1 machine ? how do you handle the load (both the connection load and the computation load since grammar and the services provided by NuGram seem pretty heavy computation-wise)? Maybe you don't need to do much given the amount of people that uses your service and I know you don't offer any guarantees with people using this service in a commercial setup but I am just wondering,

Thanks,

Dominique Boucher said...

@Alexandre I'll address those question in an upcoming post. Stay tuned!