Drupal Planet

Drupal Association blog: We asked, you answered. DrupalCon North America Location Survey Results

1 month 1 week ago

Over the past few years, we’ve been listening to the community ask for explanation as to why we haven’t had any DrupalCon North America locations outside of the United States - after all it’s called DrupalCon North America, not DrupalCon U.S.A. This isn’t something we’ve taken lightly or ignored. DrupalCon North America is a major funding source for the Drupal Association, and in that regard, a major funding source of Drupal.org and the engineering work that keeps the code accessible and available for everyone.

We’ve looked at many North American cities over the years - a lot in the United States, but some outside the U.S. also. For our 2019 and 2020 location search we directly asked several cities in Canada to bid on this event, so that we could do financial and accomodation comparisons against U.S. options. I will give you the spoiler up front: 2019 and 2020 will not be in Canada or Mexico, they will be in the United States. The cities that bid were competitive, but in the end did not prevail due to things like dates overlapping with Passover and simply not being the most effective bid in comparison to the winners.

But with these cities in mind, and the voices of the community in our ears, we decided to go deeper and explore what a Canadian or Mexican DrupalCon would look like, based on survey feedback from the community and hard numbers from our history and bids. Here is that deeper look.

First, let me say that Drupal Association staff does not think solely about finances in making these decisions. We spend a lot of time getting to know the city, the vibe, the culture and the openness to a community that celebrates diversity and has a plethora of unique needs. It’s important to you, and it’s important to us.

Let’s also acknowledge that DrupalCon North America greatly underwrites the Drupal Association work and Drupal.org infrastructure to help keep the project going. So while money is not the only thing - it is very important.

So, let’s talk about finances. There are a lot of things that go into making a DrupalCon financially viable, and we did a pretty thorough job of outlining them all in our blog series last fall dedicated to the finances of DrupalCon Europe. I suggest you take a look at those, specifically the one on Solving The Financial Problem to get a good understanding on what it takes to make DrupalCon happen. A truncated look shows that there are three (3) main aspects and goals to DrupalCon finances:

  • Expenses: everything we have to spend to make it happen
    • Goal: produce show on a tight budget
  • Revenue, attendee tickets: how many people will show up
    • Goal: people show up
  • Revenue, sponsorship commitment: how much sponsors will spend to support the event
    • Goal: sponsorships have value and continue to support us
Expenses

In a look at expenses there are a vast array of things that we spend money on - from facilities and catering to program guides and paying the person who watches coat check while you’re sprinting on Friday. And overall, the proposals we’ve received from cities within the United States and outside of the United States have been fairly competitive for expenses directly related to the venue and infrastructure. That’s awesome!

There are some other indirect expenses we consider too like cost of hotel rooms, which can greatly affect whether people can afford to stay in the city, and generally Canadian cities - for example - tend to be a bit more expensive than some of our U.S. options. Other considerations include: whether the city is a airport hub for enough domestic and international flights to get people there easily; ease of setting up foreign bank accounts or legal business statuses in specific countries in order for us to operate there (including increased staff time to do this); cost of import/export for our production gear (this applies to sponsors as well). There are workarounds for some of these, and that's what we explore during an RFP process. Based on estimates, a DrupalCon outside the United States tends to pen out to be at least 10% more expensive than one within the United States - that’s around $100,000 - $150,000.

In general, the expenses section is a place where we can explore more work-arounds and potentially find a way to make a non-U.S. DrupalCon happen. However, because of DrupalCon team capacity during 2017 (the timeframe while we were contracting 2019 - 2020 cities) this is not something we could do for the immediately upcoming DrupalCons.

Revenue

As I mentioned above, revenue from DrupalCon North America is a driving force for the Drupal Association and Drupal.org. Ensuring attendee ticket sales and sponsorship revenue remain consistent from year to year (or grow) is extremely important to helping ensure our staff are funded and Drupal.org is kept running. In order to make certain that funding holds consistent and we’re able to keep Drupal.org healthy we need to keep DrupalCon North America profit margins around roughly 30-35% per event.

Here is where things start to fall apart for non-U.S. cities in the immediate future.

To better evaluate our current and potential revenue, we created 2 surveys and put them out to the public/community to participate.

Survey targets:

  • Past and potential attendees
  • Past and existing sponsors
Revenue, Attendee Ticket Sales

DrupalCon attendees are the main audience where we hear the cry for a DrupalCon outside of the United States. Individual ticket sales make up 62% of our event revenue.

Our survey to attendees had 1258 respondents. 92% of those people have attended DrupalCon North America in the past, and 99% have attended a DrupalCon somewhere in the world. So this sample represents people who are likely to attend in the future.

Since we’re talking about Revenue, it’s important to know who is paying for these people to attend. 79% of these attendees are funded by their employers. That’s a significant number and important to think about as we move into a business case for companies to attend DrupalCon.

Next, we followed up on that. “If your employer funds your trip to DrupalCon, are they willing to pay for you to travel outside the U.S.?” Of our 79% - 38% answered “No” (this number is adjusted from the chart percentages below because the question was “IF your employer pays”, and 120 people answered that they pay for themselves). That means, of our original sample size, now only 71% of attendees are still eligible to attend (22% self-funded + (62% of 79%) = roughly 71%).

Based on the responses, our projected revenue would decrease by roughly 29%.

Revenue, Sponsorships

Sponsors provide 38% of DrupalCon revenue, their sponsorships currently underwrite the cost of early bird tickets (that’s a whole other problem), and the event would simply not happen without them. They provide the foundation for the event in financing, they are the exhibit hall, and a large portion of our attendees are sponsor company employees. If sponsors don't come, we lose money and don't achieve a key purpose of our event: connecting new business decision makers with agency owners to grow adoption.

In our survey to them, we presented a hypothetical scenario in which DrupalCon takes place in Canada.

Our leading question for sponsors was “Do you do business in Canada?” and 70% of 44 responses said “No”. This doesn’t eliminate possibility, but it is the trend for the questions that followed.

We also asked “Would you sponsor a DrupalCon in Canada at same levels as you have in the past?” and only 39% of respondents answered “Yes”.

Of these sponsors, many wrote anecdotally that they simply could not support a business case for having an event in Canada.

To Sum it Up

While we’ve had advanced talks with Canadian cities, and two were finalists for 2019 and 2020 making it past initial RFP rounds, as of now we haven’t found solutions to enough of these issues to fit a DrupalCon North America within our required profit margin.

The numbers presented by the surveys would put profit margin for a DrupalCon North America outside the U.S. at an estimated 6% profit margin and would risk actually losing money for the Drupal Association. A situation and risk we cannot allow the Association to bear.

This is disappointing for many of us - and we know it is for many of you as well. We would love to see DrupalCon North America move beyond the U.S. borders, however it will not happen until at least 2021.

In between now and our next location RFP, we will continue to look at models that might make this possible. As we explore these challenges and talk more with sponsors and cities, we will share with the community any progress or new challenges as they become relevant. We appreciate your passion on this topic and understand the concerns with hosting DrupalCon in the United States for another two (2) years, especially based in our current climate of travel restraints in to the U.S. We wish it were not difficult for our community to come together.

We appreciate everyone who took the time to participate in our surveys and were honest about their desires, motivations and realities of their travel to and participation in DrupalCon. We're excited seeing many of you in Nashville this week, and hope many of you will join us in 2019 for DrupalCon Gedfyuikemndjfkioiujhtrj - sorry, something has happened to my keyboard. ¯\_(ツ)_/¯

_________________________

We invite you to share thoughts in the comments section below on how you think DrupalCon 2019 and 2020 can help provide more opportunity for community members outside the United States to participate in the event - either through direct attendance or through virtual participation of some kind. What are your ideas?

File attachments:  Business in Canada.png Employer Fund International Travel.png WhoFunds.png Would you Sponsor.png

Dries Buytaert: Defining Drupal's values and principles

1 month 1 week ago

Since its founding, Drupal has grown a great deal, and today there are thousands of contributors and organizations that make up our community. Over the course of seventeen years, we have spent a great amount of time and effort scaling our community. As a result, Drupal has evolved into one of the largest open source projects in the world.

Today, the Drupal project serves as a role model to many other open source projects; from our governance and funding models, to how we work together globally with thousands of contributors, to our 3,000+ person conferences. However, the work required to scale our community is a continuous process.

Prompted by feedback from the Drupal community, scaling Drupal will be a key focus for me throughout 2018. I have heard a lot of great ideas about how we can scale our community, in addition to improving how we all work together. Today, I wanted to start by better establishing Drupal's values and principles, as it is at the core of everything we do.

Remarkably, after all these years, our values (what guides these behaviors) and our principles (our most important behaviors) are still primarily communicated through word of mouth.

In recent years, various market trends and challenging community events have inspired a variety of changes in the Drupal community. It's in times like these that we need to rely on our values and principles the most. However, that is very difficult to do when our values and principles aren't properly documented.

Over the course of the last five months, I have tried to capture our fundamental values and principles. Based on more than seventeen years of leading and growing the Drupal project, I tried to articulate what I know are "fundamental truths": the culture and behaviors members of our community uphold, how we optimize technical and non-technical decision making, and the attributes shared by successful contributors and leaders in the Drupal project.

Capturing our values and principles as accurately as I could was challenging work. I spent many hours writing, rewriting, and discarding them, and I consulted numerous people in the process. After a lot of consideration, I ended up with five value statements, supported by eleven detailed principles.

I shared both the values and the principles on Drupal.org as version 1.0-alpha. I labeled it alpha, because the principles and values aren't necessarily complete. While I have strong conviction in each of the Drupal principles and corresponding values, some of our values and principles are hard to capture in words, and by no means will I have described them perfectly. However, I arrived at a point where I wanted to share what I have drafted, open it up to the community for feedback, and move the draft forward more collaboratively.

While this may be the first time I've tried to articulate our values and principles in one document, many of these principles have guided the project for a very long time. If communicated well, these principles and values should inspire us to be our best selves, enable us to make good decisions fast, and guide us to work as one unified community.

I also believe this document is an important starting point and framework to help address additional (and potentially unidentified) needs. For example, some have asked for clearer principles about what behavior will and will not be tolerated in addition to defining community values surrounding justice and equity. I hope that this document lays the groundwork for that.

Throughout the writing process, I consulted the work of the Community Governance Group and the feedback that was collected in discussions with the community last fall. The 1.0-alpha version was also reviewed by the following people: Tiffany Farriss, George DeMet, Megan Sanicki, Adam Goodman, Gigi Anderson, Mark Winberry, Angie Byron, ASH Heath, Steve Francia, Rachel Lawson, Helena McCabe, Adam Bergstein, Paul Johnson, Michael Anello, Donna Benjamin, Neil Drumm, Fatima Khalid, Sally Young, Daniel Wehner and Ryan Szrama. I'd like to thank everyone for their input.

As a next step, I invite you to provide feedback. The best way to provide feedback is in the issue queue of the Drupal governance project, but there will also be opportunities to provide feedback at upcoming Drupal events, including DrupalCon Nashville.

Acro Media: Automated Drupal Code Testing: What, Why, and When To Do It

1 month 1 week ago

Automated testing is like flossing your teeth: you know you should do it, you might even tell people you do it, but chances are you don't do it nearly as often or as consistently as you ought to. Maybe you only run one automated test. On five percent of your code. Sometimes.

Manual testing vs automated testing

Manual testing—where a real live person clicks around and verifies that the code does everything it's supposed to do—has its uses. But for large-scale projects, or in cases where you need to test the same code repeatedly, automated testing can be both more cost-effective and more fruitful. You know how your eye can look at the same spelling error six times without seeing it? Automated testing can catch issues like that. It's great for rote, boring tasks that humans gloss over.

In test-driven development, you actually build the tests first and then write the code that fulfills those tests. But you don't necessarily have to do automated testing that way. Tests can be written afterwards. Sometimes it's old code that gets automated tests built for it, for instance.

Writing the test probably takes more time than it takes to initially test it. But then it's done, and you can re-run the test any time you make any change to that site. Even if you don't change anything near that code, you can run the test anyways and catch those instances where you're like, "I'm sure it won't break that"…but it does.

Drupal maintainers try to be really strict about the automated tests. Since lots of people use the modules, and lots of people contribute to them, you have all these different people submitting code. Having a good test suite can really improve your confidence in a module. If each submission comes with new tests that you can run and verify, you can be far more confident in the quality of that code.

Time savings

Say you do 10 hours of manual testing for each release. If, on the other hand, you spend 10 hours writing automated tests for each release, then for release #2, you're actually doing 20 hours of testing, and for release #3 you're doing 30 hours of testing. You're only putting in 10 man hours each time, but your test suite keeps growing, and the scope of what you're testing increases exponentially because you can rerun the same tests each time.

Why is automated testing such a hard sell?

Going back to the flossing example: why don't you floss? It takes 60 seconds. But you only really get the benefits if you do it all the time.

More to the point: not doing it takes time to become bad. Skipping automated testing on your first release is maybe not a big deal. Your code base is small, it's probably only doing a couple things, so manual testing is very feasible. But as the project grows and gets more complex, manual testing becomes increasingly unwieldy, and then you get to a point where you think, we're too far into this to add automated testing now.

But the truth is you can start at any point. It's never too late to start proactively doing things that will benefit your site.

What prevents people from getting started?

Usually the thing that stymies people is that they're not set up for automated testing: they don't have a continuous integration environment or a nice testing environment to run the tests on. Or maybe they've neglected it for so long that the regular tests don't work anymore; they write their first test and they have all these other problems because they've let things languish, so they give up.

What is continuous integration, you ask? It means every time you do a change and you push it out, it attempts to integrate it in with the whole project. It will deploy it to a server, it will compile the code if necessary, it will run code standards and automated tests and so on. When you have that stuff all set up to run automatically, you just make the code, push it to your version control, and forget about it. If something goes south, it'll send you an email saying this didn't pass. Otherwise, you don't have to think about it again.

How much of your code should be covered by automated testing?

You kind of want to be in that 95% range, although it's true that you can have parts that aren't easily testable. You can cover a lot of code, but you might still be missing use cases. Maybe you test taxes, and you test discounts, but you don't test taxes and discounts together. So technically you have full code coverage, but you're not using them in combination. So code coverage is a nice metric, but it doesn't tell the whole story.

TLDR: Automated testing is like flossing. You should really do it.

More from Acro Media Chat with us

If you'd like to talk about adding automated testing to your Drupal Commerce website, or if you just want to to discuss how Drupal Commerce fits into your ecommerce solution, give us a shout. We're happy to chat.

ComputerMinds.co.uk: Rendering Drupal 8 fields (the right way)

1 month 1 week ago

Once upon a time, we wrote an article about how to render fields on their own in Drupal 7, which was really handy because Drupal 7 wasn't always intuitive. It's common to want to display a field outside of the context of its main entity page, like showing author information in a sidebar block or in a panel, but you had to just know which functions to use. Drupal 8 has come along since then using 'grown up' things like objects and methods, which actually makes the job a little easier. So now we have this:

The short answer $node->field_my_thing->view();Quick reference

I'll cover these in detail below, but here are the things you might want to be doing:

  1. Render a field as if it were on the node/entity page (e.g. with the markup from the normal formatter and full field template)

Drupal 7: field_view_field(), optionally passing a view mode string, or formatter settings array.
Drupal 8: $node->field_my_thing->view(), optionally passing a view mode string, or formatter settings array.

  1. Just get a single value out of a field (i.e. raw values, usually as originally inputted)

Drupal 7: field_get_items() and then retrieve the index you want from the array.
Drupal 8: $node->field_my_thing->get(0)->value, passing just the index you want to get(). Properties other than 'value' may be available.

  1. Render a single value as if it were on the node/entity page (e.g. with the normal formatter's markup, but not all the wrappers that Drupal's field templates give you)

Drupal 7: field_view_value(), optionally passing a view mode string, or formatter settings array, but always passing the actual items array.
Drupal 8: $node->field_my_thing->get(0)->view(), passing just the index you want to get() and optionally passing a view mode string, or formatter settings array to view().

The long answer

Now that entities like nodes are properly classed objects, and fields use the fancy new Typed Data API, we don't need to care about the underlying data structure for nodes or their fields, we can just call the method to perform the operation we want! You know, just what methods are supposed to be for! You want to view a field? Just call its 'view' method.

The output will be automatically sanitised and goes through the normal formatter for the field, as well as the regular field template. You can specify whether you want it rendered as if it were in a teaser or other view mode, by passing in the view mode string, just as we did with field_view_field(). (Or you might have used something like $node->field_my_thing['und'][0]['value'] - in which case, go back and read our article for Drupal 7!)

$node->field_my_thing->view('teaser');

Or even override the formatter to be used altogether (which is handy if the field would normally be hidden in any view mode):

$node->field_my_thing->view([ 'type' => 'image', 'label' => 'hidden', 'settings' => array( 'image_style' => 'larger_thumbnail', 'image_link' => 'content', ), ]);

This does assume that your field ('field_my_thing') in my examples does at least exist on your node (even if it's empty). You may want to wrap the whole code in a try/catch block, just in case it might not.

For bonus points, you could load up the normal formatter settings, and tweak them:

use Drupal\Core\Entity\Entity\EntityViewDisplay; // If you have the entity/node you want, use collectRenderDisplay() as it may // already be statically cached, plus it goes through various alter hooks. $display_options = EntityViewDisplay::collectRenderDisplay($node, 'teaser') ->getComponent('field_images'); // Otherwise - or if you intentionally want to re-use the settings from another // unrelated entity type, bundle or display mode - just load the display config. $display_options = EntityViewDisplay::load('pagaraph.media_pod.teaser') ->getComponent('field_images'); // Then tweak those display options and view the field. $display_options['settings']['image_style'] = 'even_bigger_thumbnail'; $node->field_my_thing->view($display_options);

This all assumes you've at least loaded your node, or other entity. (It works with any content entity, like terms, paragraphs, etc!) You'd probably be putting the result of the view() method (which will be a render array) into a variable to be used in a twig template via a preprocess hook. Or maybe you're just adding it into an existing render array, like a custom block plugin. (Either way, you probably shouldn't then be rendering it into a string yourself, let Drupal do that for you.)

You can even just view a single item within a multi-value field like this, here using an 'if' condition to be a bit more graceful than a try/catch. This is the equivalent of using field_view_value() from Drupal 7, so also skips Drupal's full field template, though includes markup produced by the field's formatter:

// View the third value, as long as there is one. if ($third = $node->field_my_thing->get(2)) { $output = $third->view(); } else { $output = []; }

That helps you see how you might get a single value too, with the get() method, though note that it still returns a classed object. To just get a raw value, without any wrapping markup or value processing/sanitisation, you might want to use something like this, instead of Drupal 7's field_get_items() :

$text = $node->field_my_thing->get(0)->value; // If the field is just a single-value field, you can omit the get() part. $value = $node->field_single_thing->value; // Some types of field use different properties. $url = $node->field_my_link->uri; // You can use getValue() to get all the properties (e.g. text value + format). $text_values = $node->field_formatted_text->getValue(); // References can be chained! $referenced_title = $node->field_my_reference->entity->field_other_thing->value;

In Drupal 7, there was also confusion around what to do for multilingual content. In Drupal 8, as long as you've got the translation you want first, all the methods I've discussed above will get you the appropriate values for your language. To get a specific translation, use:

if ($node->hasTranslation($candidate)) { $node = $node->getTranslation($langcode); } This Rocks.

You get to use proper modern methods on a proper typed data API. Sanitisation is done for you. You don't need to care what the underlying data structure is. And you don't need to remember some magic global procedural functions, because the methods are obvious, right there on the thing you want to use them on (the field item class). If the Drupal 7 version of this was brilliant, that makes the Drupal 8 version of this even better. Brilliant-er?

OPTASY: What Are the Differences Between PHPStorm and WebStorm? Which IDE Is Right for You?

1 month 1 week ago
What Are the Differences Between PHPStorm and WebStorm? Which IDE Is Right for You? radu.simileanu Tue, 04/10/2018 - 09:38

Feeling stuck? Can't seem to put a finger on at least a few clear differences between PHPStorm and WebStorm? And you need to choose the most suitable IDE software for web development?

There sure must be some strong differences, other than:

PHPStorm doesn't provide JavaScript-oriented plugin support right out-of-the-box like WebStorm does.

Now, before we go “hunting” some key differences between PHPStorm and WebStorm, I'd like to add one last recommendation to consider when you select the right IDE for you:

It all comes down to evaluating various solutions and identifying not THE BEST, but the application's perfectly suited to your specific needs.

That being said, without further ado, let's evaluate the “candidates”!

Jeff Geerling's Blog: A modern way to build and develop Drupal 8 sites, using Composer

1 month 1 week ago

The Drupal community has been on an interesting journey since the launch of Drupal 8 in 2015. In the past three years, as the community has started to get its sea legs 'off the island' (using tools, libraries, and techniques used widely in the general PHP community), there have been growing pains.

One area where the pains have been (and sometimes still are) highly visible is in how Drupal and Composer work together. I've written posts like Composer and Drupal are still strange bedfellows in the past, and while in some ways that's still the case, we as a community are getting closer and closer to a nirvana with modern Drupal site building and project management.

For example, in preparing a hands-on portion of my and Matthew Grasmick's upcoming DrupalCon Nashville lab session on Composer and Drupal, I found that we're already to the point where you can go from literally zero to a fully functional and complete Drupal site codebase—along with a functional local development environment—in about 10 or 15 minutes:

Agiledrop.com Blog: AGILEDROP: Day zero at DrupalCon Nashville

1 month 1 week ago
It was a long trip from Slovenia to Nashville, but after a good night sleep, I headed towards Music City Center for the first day of DrupalCon. Mondays at DrupalCon are reserved for sub-events called summits. There were Decoupled Summit, Higher Education Summit, Government Summit, Media and Publishing Summit, Nonprofit Summit, Community Summit and Business Summit. The last is the one I picked to attend.    Business Summit The event was organized by Elia Albarran, Four Kitchens and Aimee Degnan, Hook 42. The idea of the event is to facilitate open discussions between Drupal agency owners… READ MORE

Nuvole: Config Distro

1 month 2 weeks ago
A new workflow for distribution configuration management.Drupal core workflow limitations

It is not a secret that the configuration management in Drupal 8 core was made for only one specific use case: move configuration from one environment to another for the same site. Everything else was left for contrib to find solutions for. Config Installer is necessary to set up a site starting from existing configuration and Config Split is needed to have environment specific configuration that go beyond simple configuration overrides, like development modules enabled only locally. But the whole workflow still assumes that developers - most often in the same team - have control over the environments and deployment of the site.

Distributions are not covered

Distributions are very different. When maintaining a distribution one doesn't develop a specific site but rather the template to build different sites with. Deploying configuration as we do it for a single site does not have any meaning because there is no one production site to deploy to. Each site based off a distribution is its own site with its own site uuid and its own possible customisations and possibly its own development and production environments. At the same time as a consumer of a distribution - a site owner of a site based off a distribution - one wants to update the distributions features while keeping the customisations.

Update hooks?

Different distributions handle this in different ways. A first approach to solve this issue is treating it the same way you would a schema update and change the configuration on a site with an update hook. The problem with this is that update hooks are meant to fix the database to be able to run with the version of code that is in use on the site. The first thing one needs to do after updating the code base is to run the update hooks to realign the database. Configuration management has different needs and when updating the configuration in an update hooks means that one has to be careful to check that the configuration is in the expected state. And the only recourse is to log and make sure the user is informed what manual actions he needs to do to update the configuration himself. As there is no place to involve the user with a choice while update hooks are run.

Config Distro

We propose a new kind of workflow, in essence it allows developers of a distribution to say how the "distributions configuration" is supposed to be updated and it allows site owners of a site based on a distribution to treat the distribution maintainers as a developer on their team. At the same time this does not interfere with the configuration management workflow for staging and deploying configuration to the production site.

The primary function of Config Distro is to allow updating a distributions configuration in a new workflow and give the site owners more control and insight on how the configuration of their site changes. It is a new way to imagine the configuration management system in a way that lets sites own their configuration but allowing distributions to provide updated configuration for existing sites.

A new dimension

In a blog post one year ago I talked about different dimensions of configuration management;

  • vertical: moving configuration between environments but for the same site
  • horizontal: moving configuration between different sites. for example for distribution updates.

Config Distro is a UI and a drush command for that workflow. It is intended to be used by site builders who base their site on a distribution and want to import configuration updates from the updated distribution. The UI is essentially a parallel to configuration import screen for the workflow supported by core. But instead of importing the configuration from the files used to move/deploy configuration between environments you import from a special configuration storage which starts out with the sites active configuration and has the distributions updates applied to it. This means you do not have problems with mismatched site uuids or things being removed that you added and were not part of the distribution. And updates can apply (or not) to translations even if the distribution did not have languages.

Where do updates come from

If you only install Config Distro, the import screen eternally shows that there is nothing to import. This is because the module only provides the framework for this to work, ie the UI and the drush command. The module is not opinionated on how or what should be updated. All the complexity can be addressed by a distribution maintainer with the help of a ConfigFilter plugin (the same way Config Split and Config Ignore work). One such module is Config Sync. All the complexity of finding out what what configuration a module has originally shipped with, what it does now and whether a user has changed the originally installed configuration is left to Config Sync and its dependencies.

Choice

Just like Config Ignore allows you to opt out of some of the configuration management workflows, Config Distro has a Config Distro Ignore module that lets you retain certain configuration from being changed when you hit the "import" button. The "Retain configuration" button is available right next to the "View differences" button.

Clicking it leads to a form that lets you choose to retain the configuration permanently or only for this specific update. It also allows you to ignore a update for a specific language.

Example

In our team we set up a site based on a distribution. We added our own login module for our single sign on and added a few webforms. Now there is a new version of the distribution with some new features and I would like upgrade the site to using the new features. I am tasked with updating the site, here is what I do:

  • I update the code of the distribution by specifying the new version in composer.json and do a composer update to get the updated code*
  • I run the database updates drush updb to align the database with the code*
  • I go to the distro update screen provided by Config Distro
    • This screen looks very familar, it looks the same way as when I import configuration changes my team mates made after I get their code with git pull.
    • I see a new module is going to be installed and a few views and settings from the distribution are updated.
    • Our webforms are not going to be removed and our custom modules not uninstalled.
  • I click "import all"
  • Now my site has the updated code and configuration so I export the configuration for deployment: drush cex and commit*
  • My colleges get the update the same way they get normal changes that happen during development*:
    • git pull get the code
    • composer install get the external code (vendor, contrib, etc)
    • drush updb align the database with the new code base.
    • drush cim import the sync configuration.

* This is the same procedure as for any module update on any site.
Distributions may add a "Auto upgrade" setting and then import the Config Distro in a post_update_hook to bypass the manual step required of site administrators upgrading the distribution.

Conclusion

Config Distro provides a new workflow for updating distributions with a familiar feel. It recognizes that update hooks are not an adequate solution for updating a sites configuration when a site owns the configuration and site administrators may have changed the initially provided configuration.
It allows developers of distributions to alter the sites configuration through ConfigFilter plugins and it gives site administrators a choice of what to import.
Config Distro is just the framework for extending cores configuration management to allow managing configuration changes to get into the site from a third party such as the distribution maintainers. It does not interfere with the traditional workflow and importing the configuration updates from a distribution should be seen as any other configuration change such as adding a new view or changing permissions: You go to an admin page and you click some things in a form, then you export the configuration and deploy it.

Future - CMI 2.0

While it already works, Config Distro is still in an alpha state. Config Distro is part of a larger effort to improve the configuration management in Drupal. You can find further information and participate in the discussion on drupal.org

Tags: Drupal 8Code Driven DevelopmentConfiguration ManagementDrupal Planet

ThinkDrop Consulting: Announcing Provision 4: Leaving the Aegir Nest

1 month 2 weeks ago
Announcing Provision 4: Leaving the Aegir Nest Jon Pugh Mon, 04/09/2018 - 11:27

As we head into DrupalCon week I've got something big to announce. With the blessing of the Aegir core maintainers, I am taking the 4.x branch of Provision I have been working on and I am separating it from the Aegir Project.

Provision 4 will still power Aegir. We are working on a patch to Hostmaster that will allow us to run a different command other than Drush, allowing Provision4 to become the primary back-end to Aegir 4. This means it will also be able to power the current generation DevShop.

Jeff Geerling's Blog: Installing PHP 7 and Composer on Windows 10

1 month 2 weeks ago

I am working a lot on Composer-based Drupal projects lately (especially gearing up for DrupalCon Nashville and my joint workshop on Drupal and Composer with Matthew Grasmick), and have been trying to come up with the simplest solutions that work across macOS, Linux, and Windows. For macOS and Linux, getting PHP and Composer installed is fairly quick and easy. However, on Windows there seem to crop up little issues here and there.

Since I finally spent a little time getting the official version of PHP for native Windows installed, I figured I'd document the process here. Note that many parts of this process were learned from the concise article Install PHP7 and Composer on Windows 10 from the website KIZU 514.

Install PHP 7 on Windows 10

Love Huria: Cool things you can do with Sass - Part 1

1 month 2 weeks ago

I have been using Sass for like past two years and now I’m a huge fan. Even though we were doing pretty much alright with writing CSS but it never gave us that kind of flexibility that Sass provides like one of the things could be managing the complexity in stylesheets as our apps get more and more substantial. Anyways, Enough about my experience already as today we have got a bunch of cool things to cover!

What is Sass?

It’s a CSS preprocessor, that’s what you will get if you start googling and its true but hold that...

Wim Leers: API-First Drupal: file uploads — 572 comments summarized

1 month 2 weeks ago

This blog post summarizes the 572 comments spanning 5 years and 2 months to get REST file upload support in #1927648 committed. Many thanks to everyone who contributed!

From February 2013 until the end of March 2017, issue #1927648 mostly … lingered. On April 3 of 2017, damiankloip posted an initial patch for an approach he’d been working on for a while, thanks to Acquia (my employer) sponsoring his time. Exactly one year later his work is committed to Drupal core. Shaped by the input of dozens of people! Just *look at that commit message!*

Background: API-First Drupal: file uploads!.

  • Little happened between February 2013 (opening of issue) and November 2015 (shipping of Drupal 8).
  • Between February 2013 and April 2014, only half a dozen comments were posted, until moshe weitzman aptly said Still a gaping hole in our REST support. Come on Internets ….
  • The first proof-of-concept patch followed in August 2014 by juampynr, but was still very rough. A fair amount of iteration occurred that month, between juampynr and Arla. It used base64 encoding, which means it needed 33% more bytes on the wire to transfer a file than if it were transmitted in binary rather than base64.
  • Then again a period of silence. Remember that this was around the time when we were trying to get Drupal 8 to a shippable state: the #1 priority was to stabilize, fix critical bugs. Not to add missing features, no matter how important. To the best of my knowledge, the funding for those who originally worked on Drupal 8’s REST API had also dried up.
  • In May 2015, another flurry of activity occurred, this time fueled by marthinal. Comment #100 was posted. Note that all patches up until this point had zero validation logic! Which of course was a massive security risk. marthinal is the first to state that this is really necessary, and does a first iteration of that.
  • A few months of silence, and then again progress in September, around DrupalCon Barcelona 2015. dawehner remarked in a review on the lack of tests for the validation logic.
  • In February 2016 I pointed out that I’m missing integration tests that prove the patch actually works. To which Berdir responded that we’d first need to figure out how to deal with File entity type access control!
  • Meanwhile, marthinal works on the integration test coverage in 2016. And … we reached comment #200.
  • In May 2016, I did a deep review, and found many problems. Quick iterations fix those problems! But then damiankloip pointed out that despite the issue being about the general File (de)serialization problem, it actually only worked for the HAL normalization. We also ended up realizing that the issue so far was about stand-alone File entity creation, even though those entities cannot be viewed stand-alone nor can they be created stand-alone through the existing Drupal UI: they can only be created to be referenced from file fields. And consequently, we have no access control logic for this yet, nor is it clear how access control should work; nor is it how validation should work! Berdir explained this well in comment 232. This lead us to explore moving parts of https://www.drupal.org/project/file_entity into core (which would be a hard blocker). The issue then went quiet again.
  • In July 2016, garphy pointed out that large file uploads still were not yet supported. Some work around that happened. In September, kylebrowning stressed this again, and provided a more detailed rationale.
  • Then … silence. Until damiankloip posted comment #281 on April 3, 2017. Acquia was sponsoring him to work on this issue. Damian is the maintainer of the serialization.module component and therefore of course wanted to see this issue get fixed. My employer Acquia agreed with my proposal to sponsor Damian to work on REST file upload support. Because after 280 comments, some fundamental capabilities are still absent: this was such a complex issue, with so many concerns and needs to balance, that it was nigh impossible to finish it without dedicated time.
    To get this going, I asked Damian to look at the documentation for a bunch of well-known sites to observe how they handle file uploads. I also asked him to read the entire issue. Combined, this should give him a good mental map of how to approach this.
  • #281 was a PoC patch that only barely worked but did support binary (non-base64) uploads. damiankloip articulated the essential things yet to be figured out: validation and access checking. Berdir chimes in with his perspective on that in #291 … in which he basically outlines what ended up in core! Besides Berdir, dagmar, dawehner, garphy, dabito, ibustos all chimed in and influenced the patch. Berdir, damiankloip and I had a meeting about how to deal with validation, and I disagreed with with both of them. And turned out to be very wrong! More feedback is provided by the now familiar names, and the intense progress/activity continues for two months, until comment #376!
  • Damian got stuck on test coverage — and since I’d written most of the REST test coverage in the preceding months, it made sense for me to pick up the baton from Damian. So I did that in July 2017, just making trivial changes that were hard to figure out. Damian then continued again, expanding test coverage and finding a core bug in the process! And so comment #400 was reached!
  • At the beginning of August, the patch was looking pretty good, so I did an architectural review. For the first time, we realized that we first needed to fix the normalization of File entities before this could land. And many more edge cases need to be tested for us to be confident that there were no security vulnerabilities. blainelang did manual testing and posted super helpful feedback based on his experience. Blaine and Damian tag-teamed for a good while, then graphy chimed in again, and we entered September. Then dawehner chimed in once more, followed by tedbow.
  • On September 6 2017, in comment #452 I marked the issue postponed on two other issues, stating that it otherwise looked tantalizingly close to RTBC. aheimlich found a problem nobody else had spotted yet, which Damian fixed.
  • Silence while the other issues get fixed … and December 21 2017 (comment #476), it finally was unblocked! Lots of detailed reviews by tedbow, gabesullice, Berdir and myself followed, as well as rerolls to address them, until I finally RTBC‘d it … in comment #502 on February 1 2018.
  • Due to the pending Drupal 8.5 release, the issue mostly sat waiting in RTBC for about two months … and then got committed on April 3 2018!!!

Damian’s first comment (preceded by many hours of research) was on April 3, 2017. Exactly one year later his work is committed to Drupal core. Shaped by the input of dozens of people! Just look at that commit message!

  • API
  • Acquia
  • Drupal

Wim Leers: API-First Drupal: file uploads!

1 month 2 weeks ago

Drupal 8’s REST API has been maturing steadily since the Drupal 8.0.0 was released in November 2015. One of the big missing features has been file upload support. As of April 3 2018, Drupal 8.6 will support it, when it ships in September 2018! See the change record for the practical consequences: https://www.drupal.org/node/2941420.

It doesn’t make sense for me to repeat what is already written in that change record: that already has both a tl;dr and a practical example.

What I’m going to do instead, is give you a high-level overview of what it took to get to this point: why it took so long, which considerations went into it, why this particular approach was chosen. You could read the entire issue (#1927648), but … it’s one of the longest issues in Drupal history, at 572 comments1. You would probably need at least an entire workday to read it all! It’s also one of the longest commit messages ever, thanks to the many, many people who shaped it over the years:

Issue #1927648 by damiankloip, Wim Leers, marthinal, tedbow, Arla, alexpott, juampynr, garphy, bc, ibustos, eiriksm, larowlan, dawehner, gcardinal, vivekvpandya, kylebrowning, Sam152, neclimdul, pnagornyak, drnikki, gaurav.goyal, queenvictoria, kim.pepper, Berdir, clemens.tolboom, blainelang, moshe weitzman, linclark, webchick, Dave Reid, dabito, skyredwang, klausi, dagmar, gabesullice, pwolanin, amateescu, slashrsm, andypost, catch, aheimlich: Allow creation of file entities from binary data via REST requests

Thanks to all of you in that commit message!

I hope it can serve as a reference not just for people interested in Drupal, but also for people outside the Drupal community: there is no One Best Practice Way to handle file uploads for RESTful APIs. There is a surprising spectrum of approaches2. Some even avoid this problem space even entirely, by only allowing to “upload” files by sending a publicly accessible URL to the file. Read on if you’re interested. Otherwise, go and give it a try!

Design rationale

General:

  • Request with Content-Type: application/octet-stream aka “raw binary” as its body, because base64-encoded means 33% more bytes, implying both slower uploads and more memory consumption. Uploading videos (often hundreds of megabytes or even gigabytes) is not really feasible with base64 encoding.
  • Request header Content-Disposition: file; filename="cat.jpg" to name the uploaded file. See the Mozilla docs. This also implies you can only upload one file per request. But of course, a client can issue multiple file upload requests in parallel, to achieve concurrent/batch uploading.
  • The two points above mean we reuse as much as possible from existing HTTP infrastructure.
  • Of course it does not make sense to have a Content-Type: application/octet-stream as the response. Usually, the response is of the same MIME type as the request. File uploads are the sensible exception.
  • This is meant for the raw file upload only; any metadata (for example: source or licensing) cannot be associated in this request: all you can provide is the name and the data for the file. To associate metadata, a second request to “upgrade” the raw file into something richer would be necessary. The performance benefit mentioned above more than makes up for the RTT of a second request in almost all cases.

PHP-specific:

  • php://input because otherwise limited by the PHP memory limit.

Drupal-specific:

  • In the case of Drupal, we know that it always represents files as File entities. They don’t contain metadata (fields), at least not with just Drupal core; it’s the file fields (@FieldType=file or @FieldType=image) that contain the metadata (because the same image may need different captions depending on its use, for example).
  • When a file is uploaded for a field on a bundle on an entity type, a File entity is created with status=false. The response contains the serialized File entity.
  • You then need a second request to make the referencing entity “use” the File entity, which will cause the File entity to get status=true.
  • Validation: Drupal core only has the infrastructure in place to use files in the context of an entity type/bundle’s file field (or derivatives thereof, such as image fields). This is why files can only be uploaded by specifying an entity type ID, bundle ID and field name: that’s the level where we have settings and validation logic in place. While not ideal, it’s pragmatic: first allowing generic file uploads would be a big undertaking and somewhat of a security nightmare.
  • Access control is similar: you need create access for the referencing entity type and field edit access for the file field.
Result

If we combine all these choices, then we end up with a new file_upload @RestResource plugin, which enables clients to upload a file:

  1. by POSTing the file’s contents
  2. to the path /file/upload/{entity_type_id}/{bundle}/{field_name}, which means that we’re uploading a file to be used by the file field of the specified entity type+bundle, and the settings/constraints of that field will be respected.
  3. … don’t forget to include a ?_format URL query argument, this determines what format the response will be in
  4. sending file data as a application/octet-stream binary data stream, that means with a Content-Type: application/octet-stream request header. (This allows uploads of an arbitrary size, including uploads larger than the PHP memory limit.)
  5. and finally, naming the file using the Content-Disposition: file; filename="filename.jpg" header
  6. the five preceding steps result in a successfully uploaded file with status=false — all that remains is to perform a second request to actually start using the file in the referencing entity!
Four years in the making — summarizing 572 comments

From February 2013 until the end of March 2017, issue #1927648 mostly … lingered. On April 3 of 2017, damiankloip posted an initial patch for an approach he’d been working on for a while, thanks to Acquia (my employer) sponsoring his time. Exactly one year later his work is committed to Drupal core. Shaped by the input of dozens of people! Just look at that commit message!

Want to actually read a summary of those 572 comments? I got you covered!

  1. It currently is the fifth longest Drupal core issue of all time! The first page, with ~300 comments, is >1 MB of HTML. ↩︎

  2. Examples: Contentful, Twitter, Dropbox and others↩︎

  • API
  • Acquia
  • Drupal

Another Drop in the Drupal Sea: Migrating from Drupal 6 to Drupal 8

1 month 2 weeks ago
Migrating from Drupal 6 to Drupal 8 Marc Sun, 04/08/2018 - 11:31am

Well, I've finally done it! I migrated this blog from Drupal 6 to Drupal 8. I did a test run yesterday with a personal blog of mine and found the process was relatively easy. Both sites are relatively simple.

There are other blog posts about the process as well as documentation on Drupal.org so I won't repeat lots of details.

I'm running Drupal 8.5.1 as of this blog post. I chose to use all of the various migration modules that come with Drupal 8 core, including the two marked as experimental. Once I had them enabled I clicked the link to get to the upgrade form in the UI. One of the sites did have file uploads and all of them were pulled in seamlessly. I had created a backup of the sites/[site-name] directory and placed it in my new Drupal 8 sites directory.

Here are issues I encountered:

  • Administration Menu is (apparently) not properly ported to Drupal 8 and it blew things up on the site yesterday. I had to manually clean things up in the database so that module was not enabled in my Drupal 8 site and clear cache.
  • The taxonomy term reference to the Tags vocabulary needed the field setting updated so that it selected the Tags vocabulary.
  • When I click the user in the Toolbar it does not switch the tray so that it shows View profile, Edit profile and Log out. (That's still an issue as I write this. I haven't investigated it enough to figure out what's going wrong, nor have I filed an issue.)
  • The feed that taxonomy provides for terms has changed slightly. I filed an issue to get Planet Drupal to use the new feed.
  • No views were imported.
  • The pathauto patterns weren't imported.
  • Disqus doesn't handle the migration.
  • Other than those things, I can't say I ran into much of anything else. And, aside from the site blowing up from Administration Menu, there wasn't much that presented a real challenge.

If you are reading this post on Planet Drupal, then you know I'm back up and running!

I'll still need to theme this site again. And I'll need to replace some functionality that was previously provided by Advanced Blog. I haven't yet installed and tested out the Drupal 8 port of Tagadelic.

One more note: I decided to just delete the comments that I had for the Drupal 6 version of this site since I don't want to use the Comment module, preferring to use Disqus instead.

Comments

Drupal core announcements: Core topic discussions at DrupalCon Nashville

1 month 2 weeks ago

DrupalCon Nashville includes a full track of core conversations where you can learn about current topics in Drupal core development, and a week of sprints where you can participate in shaping Drupal's future.

In addition to the core conversations, we have a few meetings on specific topics for future core development. These meetings will be very focused, so contact the listed organizer for each if you are interested in participating. There are also birds-of-a-feather (BoF) sessions, which are open to all attendees without notice.

Also be sure to watch Dries' keynote for ideas about Drupal's future! Check out the extended Dries Q&A session on Thursday as well to get even more questions answered.

Time Topic Organizer Monday, 9 April, 10:00 Configuration validation to support REST and JS Wim Leers Tuesday, 10 April, 10:45 Improving Drupal's evaluator experience (BoF) tedbow Tuesday, 10 April, 15:45 Layout Initiative meeting tim.plunkett Wednesday, 11 April, 10:45 Official local development environment (BoF) tedbow Wednesday, 11 April, 14:15 Media roadmap meeting phenaproxima Friday, 13 April, 09:00 Release cycle changes discussion (only core committers) Gábor Hojtsy Friday, 13 April, 11:00 Automated security updates hestenet
Checked
2 days 2 hours ago
Drupal.org - aggregated feeds in category Planet Drupal
Subscribe to Drupal Planet feed