424 stories
·
13 followers

One SQL to rule them all: an efficient and syntactically idiomatic approach to management of streams and tables

2 Shares

One SQL to rule them all: an efficient and syntactically idiomatic approach to management of streams and tables Begoli et al., SIGMOD’19

In data processing it seems, all roads eventually lead back to SQL! Today’s paper choice is authored by a collection of experts from the Apache Beam, Apache Calcite, and Apache Flink projects, outlining their experiences building SQL interfaces for streaming. The net result is a set of proposed extensions to the SQL standard itself, being worked on under the auspices of the international SQL standardization body.

The thesis of this paper, supported by experience developing large open-source frameworks supporting real-world streaming use cases, is that the SQL language and relational model as-is and with minor non-intrusive extensions, can be very effective for manipulation of streaming data.

Many of the ideas presented here are already implemented by Apache Beam, Calcite, and Flink in some form, as one option amongst several. The streaming SQL interface has been adopted by Alibaba, Hauwei, Lyft, Uber and others, with the following feedback presented to the authors as to why they made this choice:

  • Development and adoption costs are significantly lower compared to non-declarative stream processing APIs
  • Familiarity with standard SQL eases adoption compared to non-standardized query languages
  • Common stream processing tasks such as windowed aggregations and joins can be easily expressed and efficiently executed due to event time semantics
  • In case of faulty application logic or service outages, a recorded data stream can be reprocessed by the same query that processes the live data stream.

Foundational principles

The big goal is to unify the abstractions of tables and streams in one common framework:

Combined, tables and streams cover the critical spectrum of business operations ranging from strategic decision making supported by historical data to near- and real-time data used in interactive analysis… We believe, based on our experience and nearly two decades of research on streaming SQL extensions, that using the same SQL semantics in a consistent manner is a productive and elegant way to unify these two modalities of data…

As the authors point out, there has been a lot of prior work in this space over many years, and the proposals presented in this paper draw on much of it. At the sharp end, they are based on lessons learned working an Apache Flink, Beam, and Calcite.

The thing that streaming adds to a traditional relational view is the concept of time. Note that a mutable database table, as perceived by a consumer across multiple queries, is already a time-varying relation (TVR). It’s just that for any one query the results always show the relation at a single point in time.

A time-varying relation is exactly what the name implies: a relationship whose contents may vary over time… The key insight, stated but under-utilized in prior work, is that streams and tables are two representations for one semantic object.

The TVR, by definition, supports the entire suite of relational operators, even in scenarios involving time-varying relational data. So the first part of the proposal is essentially a no-op! We want TVRs, and that’s what relations already are, so let’s just use them – and make it explicit that SQL operates over TVRs as we do so.

We do need some extensions to deal with the notion of event time though. In particular, we need to take care to separate the event time from the processing time (which could be some arbitrary time later). We also need to understand that events will not necessarily be presented for processing in event-time order.

We propose to support event time semantics via two concepts: explicit event timestamps and watermarks. Together, these allow correct event time calculation, such as grouping into intervals (or windows) of event time, to be effectively expressed and carried out without consuming unbounded resources.

The watermarking model used traces its lineage back to Millwheel, Google Cloud Dataflow, and from there to Beam and Flink. For each moment in processing time, the watermark specifies the event timestamp up to which the input is believed to be complete at that point in processing time.

The third piece of the puzzle is to provide some control over how relations are rendered and when rows are materialized. For example: should a query’s output change instantaneously to reflect any new input (normally overkill), or do we just want to see batched updates at the end of a window?

Example

Query 7 from the NEXmark stream querying benchmark monitors the highest price items in an auction. Every ten minutes, it returns the highest bid and associated itemid for the most recent ten minutes.

Here’s what it looks like expressed using the proposed SQL extensions. Rather than give a lengthy prose description of what’s going on, I’ve chosen just to annotate the query itself. Hopefully that’s enough for you to get the gist…

Given the following events

Then a query evaluated at 8:21 would yield the following TVR:

Whereas an evaluation at 8:13 would have looked different:

Note that as currently expressed, the query returns point in time results, but we can use the materialisation delay extensions to change that if we want to. For example, SELECT ... EMIT AFTER WATERMARK; will only emit rows once the watermark has passed the end of the a window.

So at 8:16 we’d see

And at 8:21:

If we want to see rows for windows regardless of watermark, but only get periodic aggregated snapshots we can use SELECT ... EMIT STREAM AFTER DELAY (STREAM here indicates we want streamed results too).

SQL Extensions

Hopefully that’s given you a good flavour. As it stands, the proposal contains 7 extensions to standard SQL:

  1. Watermarked event time column: an event time column in a relation is a distinguished column of type TIMESTAMP with an associated watermark. The watermark is maintained by the system.
  2. Grouping on event timestamps: when a GROUP BY clause groups on an event time column, only groups with a key less than the watermark for the column are included
  3. Event-time windowing functions: starting with Tumble and Hop which take a relation and event time column descriptor and return a relation with additional event-time columns as output. Tumble produces equally spaced disjoint windows, Hop produces equally sized sliding windows.
  4. Stream materialization: EMIT STREAM results in a time-varying relation representing changes to the classic result of the query. Additional columns indicate whether or not the row is a retraction of the previous row, the changelog processing time offset of the row, and a sequence number relative to other changes to the same event time grouping.
  5. Materialization delay: when a query has an EMIT AFTER WATERMARK modifier, only complete rows from the results are materialized
  6. Periodic materialization: when a query has EMIT AFTER DELAY d rows are materialized with period d, instead of continuously.
  7. Combined materialization delay: when a query has EMIT AFTER DELAY d AND AFTER WATERMARK rows are materialized with period d as well as when complete.

Hop example

Emit Stream example

Lessons learned along the way

Section 5 in the paper contains a list of lessons learned from Apache Calcite, Flink, and Beam that have informed the design. I don’t have the space to cover them all here, but as a flavour here are a couple that caught my eye:

  • Because event timestamps are just regular attributes and can be referenced in ordinary expressions, it’s possible that an expression result may not remain aligned with watermarks, which needs to be taken into account during query planning.
  • Users find it hard to reason about the optimal use of event time in queries, which can lead to expensive execution plans with undesirable semantics.

Future work

For me, the impressive thing about this work is how much can be achieved with so little. A read through the future work section though (§8) soon shows that the set of extensions is going to need to grow some more before this is done.

As an example, one area that caught my eye was the SQL standard definition that time in SQL queries is fixed at query time (either to the current time, or to a specified fixed time using AS OF SYSTEM TIME). That means you can’t yet express a view over the tail of a stream (you can use an expression such as CURRENT_TIME - INTERVAL ‘1’ HOUR in a predicate, but CURRENT_TIME takes on one fixed value at the time the query is executed). Likewise enriching a TVR with attributes from a temporal table at a specific point in time (e.g. event time) needs additional support.



Read the whole story
graydon
101 days ago
reply
Share this story
Delete

Towards multiverse databases

4 Shares

Towards multiverse databases Marzoev et al., HotOS’19

A typical backing store for a web application contains data for many users. The application makes queries on behalf of an authenticated user, but it is up to the application itself to make sure that the user only sees data they are entitled to see.

Any frontend can access the whole store, regardless of the application user consuming the results. Therefore, frontend code is responsible for permission checks and privacy-preserving transformations that protect user’s data. This is dangerous and error-prone, and has caused many real-world bugs… the trusted computing base (TCB) effectively includes the entire application.

The central idea behind multiverse databases is to push the data access and privacy rules into the database itself. The database takes on responsibility for authorization and transformation, and the application retains responsibility only for authentication and correct delegation of the authenticated principal on a database call. Such a design rules out an entire class of application errors, protecting private data from accidentally leaking.

It would be safer and easier to specify and transparently enforce access policies once, at the shared backend store interface. Although state-of-the-are databases have security features designed for exactly this purpose, such as row-level access policies and grants of views, these features are too limiting for many web applications.

In particular, data-dependent privacy policies may not fit neatly into row- or column-level access controls, and it may be permissible to expose aggregate or transformed information that traditional access control would prevent.

With multiverse databases, each user sees a consistent “parallel universe” database containing only the data that user is allowed to see. Thus an application can issue any query, and we can rest safe in the knowledge that it will only see permitted data.

The challenging thing of course, is efficiently maintaining all of these parallel universes. We’ll get to that, but first let’s look at some examples of privacy policies and how they can be expressed.

Expressing privacy policies

In the prototype implementation, policies are expressed in a language similar to Google Cloud Firestore security rules. A policy just needs to be a deterministic function of a given update’s record data and the database contents. Today the following are supported:

  • Row suppression policies (e.g. exclude rows matching this pattern)
  • Column rewrite policies (e.g. translate / mask values)
  • Group policies, supporting role-based (i.e., data-dependent access controls)
  • Aggregation policies, which restrict a universe to see certain tables or columns only in aggregated or differentially private form.

Consider a class discussion forum application (e.g. Piazza) in which students can post questions that are anonymous to other students, but not anonymous to instructors. We can express this policy with a combination of row suppression and column rewriting:

Maybe we want to allow teaching assistants (TAs) to see anonymous posts in the classes they teach. We can define a group via a membership condition and then attach policies to that group:

Write policies (not supported in the current implementation) permit specification of allowed updates. For example:

An aggregation policy could be used to rewrite any matching aggregation into a differentially-private version. The basis for this could be e.g. Chan et al.’s ‘Private and continual release of statistics’. Composing such policies with other policies remains an open research question.

Managing universes

A multiverse database consists of a base universe, which represents the database without any read-side privacy policies applied, and many user universes, which are transformed copies of the database.

For good query performance we’d like to pre-compute these per-user universes. If we do that naively though, we’re going to end up with a lot of universes to store and maintain and the storage requirements alone will be prohibitive.

A space- and compute-efficient multiverse database clearly cannot materialize all user universes in their entirety, and must support high-performance incremental updates to the user universes. It therefore requires partially-materialized views that support high-performance updates. Recent research has provided this missing key primitive. Specifically, scalable, parallel streaming dataflow computing systems now support partially-stateful and dynamically-changing dataflows. These ideas make an efficient multiverse database possible.

So, we make the database tables in the base universe be the root vertices of a dataflow, and as the base universe is updated records move through the flow into user universes. Where an edge in the dataflow graph crosses a universe boundary, any necessary dataflow operators to enforce the required privacy policies are inserted. All applicable policies are applied on every edge that transitions into a given user universe, so whichever path data takes to get there we know the policies will have been enforced.

We can build the dataflow graph up dynamically, extending the flow’s for a user’s universe the first time a query is executed. The amount of computation required on a base update can be reduced by sharing computation and cached data between universes. Implementing this as a joint partially-stateful dataflow is the key to doing this safely.

By reasoning about all users’ queries as a joint dataflow, the system can detect such sharing: when identical dataflow paths exist, they can be merged.

Logically distinct, but functionally equivalent dataflow vertices can also share a common backing store. Any record reaching such a vertex in a given universe implies that universe has access to it, so the system can safely expose the shared copy.

Just as user universes can be created on demand, so inactive universes can be destroyed on demand as well. Under the covers, these are all manipulations of the dataflow graph, which partially-stateful dataflow can support without downtime.

Prototype evaluation

The authors have built a prototype implementation of these ideas based on the Noria dataflow engine. It runs to about 2,000 lines of Rust. A Piazza-style class forum discussion application with 1M posts, 1,000 classes, and a privacy policy allowing TAs to see anonymous posts is used as the basis for benchmarking.

The team compare the prototype with 5,000 active user universes, a MySQL implementation with inlined privacy policies (‘with AP’) and a MySQL implementation that does not enforce the privacy policy (‘without AP’):

Since the prototype is serving reads from a pre-computed universe stored in memory cached results are fast and make for a very favourable comparison against MySQL. Writes are significantly slower though (about 2x) – much of this overhead is in the implementation rather than essential. Memory footprint is 0.5GB with one universe, and 1.1GB with 5,000 universes, introduces a shared record store for identical queries reduces their space footprint by 94%.

These results are encouraging, but a realistic multiverse database must further reduce memory overhead and efficiently run millions of user universes across machines. Neither Noria nor any other current dataflow system support execution of the huge dataflows that such a deployment requires. In particular, changes to the dataflow must avoid full traversals of the dataflow graph for faster universe creation.

Support for write authorization policies (with some tricky consistency considerations for data-dependent policies) is future work, as is the development of a policy-checker (perhaps similar to Amazon’s SMT-based policy checker for AWS) to help ensure policies themselves are consistent and complete.

Our initial results indicate that a large, dynamic, and partially-stateful dataflow can support practical multiverse databases that are easy to use and achieve good performance and acceptable overheads. We are excited to further explore the multiverse database paradigm and associated research directions.



Read the whole story
zwol
116 days ago
reply
Pittsburgh, PA
graydon
117 days ago
reply
Share this story
Delete

Introducing PlaNet: A Deep Planning Network for Reinforcement Learning

3 Shares


Research into how artificial agents can improve their decisions over time is progressing rapidly via reinforcement learning (RL). For this technique, an agent observes a stream of sensory inputs (e.g. camera images) while choosing actions (e.g. motor commands), and sometimes receives a reward for achieving a specified goal. Model-free approaches to RL aim to directly predict good actions from the sensory observations, enabling DeepMind's DQN to play Atari and other agents to control robots. However, this blackbox approach often requires several weeks of simulated interaction to learn through trial and error, limiting its usefulness in practice.

Model-based RL, in contrast, attempts to have agents learn how the world behaves in general. Instead of directly mapping observations to actions, this allows an agent to explicitly plan ahead, to more carefully select actions by "imagining" their long-term outcomes. Model-based approaches have achieved substantial successes, including AlphaGo, which imagines taking sequences of moves on a fictitious board with the known rules of the game. However, to leverage planning in unknown environments (such as controlling a robot given only pixels as input), the agent must learn the rules or dynamics from experience. Because such dynamics models in principle allow for higher efficiency and natural multi-task learning, creating models that are accurate enough for successful planning is a long-standing goal of RL.

To spur progress on this research challenge and in collaboration with DeepMind, we present the Deep Planning Network (PlaNet) agent, which learns a world model from image inputs only and successfully leverages it for planning. PlaNet solves a variety of image-based control tasks, competing with advanced model-free agents in terms of final performance while being 5000% more data efficient on average. We are additionally releasing the source code for the research community to build upon.
The PlaNet agent learning to solve a variety of continuous control tasks from images in 2000 attempts. Previous agents that do not learn a model of the environment often require 50 times as many attempts to reach comparable performance.
How PlaNet Works
In short, PlaNet learns a dynamics model given image inputs and efficiently plans with it to gather new experience. In contrast to previous methods that plan over images, we rely on a compact sequence of hidden or latent states. This is called a latent dynamics model: instead of directly predicting from one image to the next image, we predict the latent state forward. The image and reward at each step is then generated from the corresponding latent state. By compressing the images in this way, the agent can automatically learn more abstract representations, such as positions and velocities of objects, making it easier to predict forward without having to generate images along the way.
Learned Latent Dynamics Model: In a latent dynamics model, the information of the input images is integrated into the hidden states (green) using the encoder network (grey trapezoids). The hidden state is then projected forward in time to predict future images (blue trapezoids) and rewards (blue rectangle).
To learn an accurate latent dynamics model, we introduce:
  • A Recurrent State Space Model: A latent dynamics model with both deterministic and stochastic components, allowing to predict a variety of possible futures as needed for robust planning, while remembering information over many time steps. Our experiments indicate both components to be crucial for high planning performance.
  • A Latent Overshooting Objective: We generalize the standard training objective for latent dynamics models to train multi-step predictions, by enforcing consistency between one-step and multi-step predictions in latent space. This yields a fast and effective objective that improves long-term predictions and is compatible with any latent sequence model.
While predicting future images allows us teach the model, encoding and decoding images (trapezoids in the figure above) requires significant computation, which would slow down planning. However, planning in the compact latent state space is fast since we only need to predict future rewards, and not images, to evaluate an action sequence. For example, the agent can imagine how the position of a ball and its distance to the goal will change for certain actions, without having to visualize the scenario. This allows us to compare 10,000 imagined action sequences with a large batch size every time the agent chooses an action. We then execute the first action of the best sequence found and replan at the next step.
Planning in Latent Space: For planning, we encode past images (gray trapezoid) into the current hidden state (green). From there, we efficiently predict future rewards for multiple action sequences. Note how the expensive image decoder (blue trapezoid) from the previous figure is gone. We then execute the first action of the best sequence found (red box).
Compared to our preceding work on world models, PlaNet works without a policy network -- it chooses actions purely by planning, so it benefits from model improvements on the spot. For the technical details, check out our online research paper or the PDF version.

PlaNet vs. Model-Free Methods
We evaluate PlaNet on continuous control tasks. The agent is only given image observations and rewards. We consider tasks that pose a variety of different challenges:
  • A cartpole swing-up task, with a fixed camera, so the cart can move out of sight. The agent thus must absorb and remember information over multiple frames.
  • A finger spin task that requires predicting two separate objects, as well as the interactions between them.
  • A cheetah running task that includes contacts with the ground that are difficult to predict precisely, calling for a model that can predict multiple possible futures.
  • A cup task, which only provides a sparse reward signal once a ball is caught. This demands accurate predictions far into the future to plan a precise sequence of actions.
  • A walker task, in which a simulated robot starts off by lying on the ground, and must first learn to stand up and then walk.
PlaNet agents trained on a variety of image-based control tasks. The animation shows the input images as the agent is solving the tasks. The tasks pose different challenges: partial observability, contacts with the ground, sparse rewards for catching a ball, and controlling a challenging bipedal robot.
Our work constitutes one of the first examples where planning with a learned model outperforms model-free methods on image-based tasks. The table below compares PlaNet to the well-known A3C agent and the D4PG agent, that combines recent advances in model-free RL. The numbers for these baselines are taken from the DeepMind Control Suite. PlaNet clearly outperforms A3C on all tasks and reaches final performance close to D4PG while, using 5000% less interaction with the environment on average.
One Agent for All Tasks
Additionally, we train a single PlaNet agent to solve all six tasks. The agent is randomly placed into different environments without knowing the task, so it needs to infer the task from its image observations. Without changes to the hyper parameters, the multi-task agent achieves the same mean performance as individual agents. While learning slower on the cartpole tasks, it learns substantially faster and reaches a higher final performance on the challenging walker task that requires exploration.
Video predictions of the PlaNet agent trained on multiple tasks. Holdout episodes collected with the trained agent are shown above and open-loop agent hallucinations below. The agent observes the first 5 frames as context to infer the task and state and accurately predicts ahead for 50 steps given a sequence of actions.
Conclusion
Our results showcase the promise of learning dynamics models for building autonomous RL agents. We advocate for further research that focuses on learning accurate dynamics models on tasks of even higher difficulty, such as 3D environments and real-world robotics tasks. A possible ingredient for scaling up is the processing power of TPUs. We are excited about the possibilities that model-based reinforcement learning opens up, including multi-task learning, hierarchical planning and active exploration using uncertainty estimates.

Acknowledgements
This project is a collaboration with Timothy Lillicrap, Ian Fischer, Ruben Villegas, Honglak Lee, David Ha and James Davidson. We further thank everybody who commented on our paper draft and provided feedback at any point throughout the project.




Read the whole story
graydon
236 days ago
reply
Share this story
Delete

All the Bad Things About Uber and Lyft In One Simple List

4 Comments and 13 Shares

Here’s the latest evidence that Uber and Lyft are destroying our world: Students at the University of California Los Angeles are taking an astonishing 11,000 app-based taxi trips every week that begin and end within the boundaries of the campus.

The report in the Daily Bruin revealed anew that Uber, Lyft, Via and the like are massively increasing car trips in many of the most walkable and transit friendly places in U.S.

It comes after a raft of recent studies have found negative effects from Uber and Lyft, such as increased congestion, higher traffic fatalities, huge declines in transit ridership and other negative impacts. It’s becoming more and more clear that Uber and Lyft having some pretty pernicious effects on public health and the environment, especially in some of the country’s largest cities.

We decided to compile it all into a comprehensive list, and well, you judge for yourself. Here we go:

They increase driving — a lot

The U.C.L.A. trips are an example of what is happening at a much wider scale: A lot more driving.

Uber and Lyft, for example, are providing 90,000 rides a day in Seattle now. That’s more than are carried daily by the city’s light rail system, the Seattle Times reports.

One study estimated that in cities with the highest Uber and Lyft adoption rates, driving has increased about 3 percent compared to the cities with the lowest. That’s an enormous amount of miles.

And transportation consultant Bruce Schaller estimates that the app-based taxis have added 5.7 billion driving miles in the nine major cities they primarily operate. (For comparison, in their first year of deployment across the U.S., e-scooters operated by private tech firms carried between 60-80 million trips.)

By the end of this year, Schaller has estimated all taxi ridership will surpass the number of trips made on buses the U.S.

uber ad

The promise of companies such as Uber and Lyft was that they would “free” city dwellers to sell their cars or not acquire them in the first place. And car ownership has declined among higher wage earners.

But a University of Chicago study found the presence of Uber and Lyft in cities actually increases new vehicle registrations. That’s because the companies encourage lower-income people to purchase cars, even advertising in some markets how people should put that new car to use — as an Uber.

They spend half their time ‘deadheading’

For every mile a Uber or Lyft car drives with a passenger, it cruises as many miles — if not more — without a passenger, a practice known in the industry as “deadheading.” Estimates of total deadheading time vary from 30 percent to as much as 60 percent.

Uber and Lyft’s policies make this worse by encouraging drivers to constantly circle to reduce wait times for users, according to John Barrios, the researcher at the University of Chicago, who has studied Uber and Lyft.

They operate in transit-friendly areas

Transit systems around the nation are losing riders to Uber and Lyft, which suggests that the companies are merely showing the need to beef up transit service across the country.

But if you drill down, something else is at work because Uber and Lyft primarily operate in areas that are best served by transit. For example in Seattle, about half the rides taken in Uber and Lyft originate in just four neighborhoods: downtown, Belltown, South Lake Union and Capitol Hill, according to David Gutman at the Seattle Times. These are some of the city’s most walkable and transit-friendly areas.

Moreover, according to Schaller, about 70 percent of Uber and Lyft trips take place in just nine American cities: Boston, Chicago, Los Angeles, Miami, New York, Philadelphia, San Francisco, Seattle and Washington, D.C.

Meanwhile, traditional taxi service, Schaller estimates, still serves more total trips in suburban and rural areas than the Ubers and Lyfts.

Why would Uber and Lyft use be so high in dense, transit-rich areas? Studies aren’t conclusive, but on average, Uber and Lyft riders, not surprisingly, skew rich and skew young.

In the top nine cities for Uber and Lyft people with incomes above $200,000 are by far the most likely to use the service. Lower-income people without cars in some less urban markets do use Uber and Lyft, but their use is dwarfed by those with high incomes, Schaller finds.

They mostly replace biking, walking or transit trips

In an ideal world, Uber and Lyft would be making good on their promise to reduce private car ownership because city dwellers would feel more comfortable selling their cars, thanks to the presence of Uber and Lyft.

But the data shows that Uber and Lyft mostly “free” people from walking and transit.

A survey of 944 Uber and Lyft riders by the Metropolitan Area Planning Council in Boston last year, found that 42 percent of riders would have taken transit if the services hadn’t been available. Another 12 percent (like those U.C.L.A students and their 11,000 on-campus taxi rides per week) said they would have biked or walked their journey. Another 5 percent would have just avoided the trip altogether.

Only about 17 percent — less than one in five — said they would have made the journey in a private car otherwise. (The remainder said they would have used a traditional taxi.)

Uber and Lyft just aren’t competitive price-wise with private car ownership, Schaller said, except in areas with expensive parking. Even with Uberpool and other shared services — which account for a small share of total business, Schaller says — Uber and Lyft increase car miles on urban streets. For each mile of driving removed, they add about 2.6 miles, he estimates.

They hurt transit 

Uber and Lyft are just crushing transit service in the U.S. A recent study estimated, for example, they had reduced bus ridership in San Francisco, for example, 12 percent since 2010 — or about 1.7 percent annually. And each year the services are offered, the effect grows, researcher Gregory Erhardt found.

Every person lured from a bus or a train into a Lyft or Uber adds congestion to the streets and emissions to the air. Even in cities that have made tremendous investments in transit — like Seattle which is investing another $50 billion in light rail — Uber and Lyft ridership recently surpassed light rail ridership.

Transit agencies simply cannot complete with private chauffeur service which is subsidized at below real costs by venture capitalists. And maybe that’s the point.

Erhardt, for example, estimated that San Francisco would have had to increase transit service 25 percent overall just to neutralize the effect of Uber and Lyft.

Worse is the tale of two cities effect: Relatively well off people in Ubers congesting the streets of Manhattan and San Francisco slow down buses full of relatively low-income people. By giving people who can afford it escape from the subway, Uber and Lyft also reduce social interaction between people of different classes and lead to a more stratified society.

They reduce political support for transit

As an added kick in the shins, Uber and Lyft degrade political support for transit. If relatively well-to-do people can hop in an Uber or a Lyft every time the bus or train is late, the political imperative to address the problem is reduced. The wealthier people substituting Uber and Lyft for transit trips have disproportionate political influence.

Cities are already capitulating. Last week, Denver partnered with Uber in a last-ditch effort to win back some riders who had jumped to the app.

In addition, right-wing ideologues have argued that Uber and Lyft make transit investment unnecessary.

They increase traffic fatalities

The University of Chicago study mentioned earlier estimated that Uber and Lyft increased traffic fatalities last year by an astonishing 1,100 — an enormous human toll. The study also found, surprisingly, that Uber and Lyft have no effect on drunk driving.

In addition, Uber and Lyft require basically no safety training for their drivers at all. In fact, the presence of these companies has motivated cities like Toronto to eliminate safety training requirements the city previous required for taxi drivers, in order to ostensibly level the playing field.

They hoard their data

One qualification with this list: Much of the information we have about Lyft and Uber is imperfect. The two companies make it difficult to study the social impacts of their activities because they jealously guard their data.

Last year, when Barrios released a study showing a lot of negative impacts from Uber and Lyft, Lyft corporate attacked the study calling it “deeply flawed.”

But Barrios had to use Google search numbers to estimate Uber and Lyft penetration in certain markets because even academic researchers don’t have access to Uber and Lyft’s raw trip data. If Uber and Lyft are honest in their denials, releasing their data could help disprove it. But so far, they have mostly refused.

Oh, and one more thing…

These are just the transportation related drawbacks. To say nothing of these companies treatment of their employees, or the behavior of their top management or their huge financial losses.

Read the whole story
popular
251 days ago
reply
38 days ago
https://keramatzade.com/Earn-wealth-with-amazing-business-ideals https://keramatzade.com/Law-of-Attraction-of-Wealth https://keramatzade.com/Ways-to-make-money https://modirebimeh.ir/online-calculation-of-iranian-life-insurance/ https://modirebimeh.ir/engineers-professional-liability-insurance/ https://modirebimeh.ir/third-party-insurance-calculation/ https://modirebimeh.ir/iran-liability-insurance-have-you-not-yet-insured-your-business-with-iran-liability-insurance/ https://modirebimeh.ir/iran-life-insurance-ganji-for-the-future-of-children-and-families/ https://modirebimeh.ir/iran-car-body-insurance-the-best-and-most-prestigious-in-the-iranian-insurance-industry/ https://modirebimeh.ir/the-most-reliable-and-unrivaled-third-party-car-insurance-in-iran/ https://keramatzade.com/14-ways-to-increase-revenue https://keramatzade.com/8-ways-to-increase-revenue https://keramatzade.com/25-jobs-with-which-you-can-earn-up-to-a-million-dollars https://keramatzade.com/success-secret-1 https://keramatzade.com/Make-Money-Online-Effective-step-by-step-money-making-techniques https://keramatzade.com/Make-money-at-home https://keramatzade.com/Ways-to-make-money-without-capital https://keramatzade.com/Creative-Money-Making-Ideas https://keramatzade.com/The-law-of-attracting-money https://keramatzade.com/Ways-to-Make-Money-at-Home https://keramatzade.com/Immediate-absorption-of-wealth-in-10-minutes-and-attractive-ways-to-get-rich https://keramatzade.com/The-secret-of-attracting-money-in-Iran-to-achieve-creative-money-maker-ideas https://keramatzade.com/Ways-to-get-rich-in-Iran-with-the-most-wonderful-business-ideas https://keramatzade.com/Astonishing-economic-intelligence-test-to-increase-financial-intelligence
sarcozona
251 days ago
reply
graydon
252 days ago
reply
acdha
252 days ago
reply
Washington, DC
Share this story
Delete
4 public comments
esran
249 days ago
reply
Ways uber/lyft services are bad, above and beyond the obvious ones.
Bristol, UK
38 days ago
https://keramatzade.com/Earn-wealth-with-amazing-business-ideals https://keramatzade.com/Law-of-Attraction-of-Wealth https://keramatzade.com/Ways-to-make-money https://modirebimeh.ir/online-calculation-of-iranian-life-insurance/ https://modirebimeh.ir/engineers-professional-liability-insurance/ https://modirebimeh.ir/third-party-insurance-calculation/ https://modirebimeh.ir/iran-liability-insurance-have-you-not-yet-insured-your-business-with-iran-liability-insurance/ https://modirebimeh.ir/iran-life-insurance-ganji-for-the-future-of-children-and-families/ https://modirebimeh.ir/iran-car-body-insurance-the-best-and-most-prestigious-in-the-iranian-insurance-industry/ https://modirebimeh.ir/the-most-reliable-and-unrivaled-third-party-car-insurance-in-iran/ https://keramatzade.com/14-ways-to-increase-revenue https://keramatzade.com/8-ways-to-increase-revenue https://keramatzade.com/25-jobs-with-which-you-can-earn-up-to-a-million-dollars https://keramatzade.com/success-secret-1 https://keramatzade.com/Make-Money-Online-Effective-step-by-step-money-making-techniques https://keramatzade.com/Make-money-at-home https://keramatzade.com/Ways-to-make-money-without-capital https://keramatzade.com/Creative-Money-Making-Ideas https://keramatzade.com/The-law-of-attracting-money https://keramatzade.com/Ways-to-Make-Money-at-Home https://keramatzade.com/Immediate-absorption-of-wealth-in-10-minutes-and-attractive-ways-to-get-rich https://keramatzade.com/The-secret-of-attracting-money-in-Iran-to-achieve-creative-money-maker-ideas https://keramatzade.com/Ways-to-get-rich-in-Iran-with-the-most-wonderful-business-ideas https://keramatzade.com/Astonishing-economic-intelligence-test-to-increase-financial-intelligence
DGA51
250 days ago
reply
Is there any positive?
Central Pennsyltucky
acdha
250 days ago
Yes: they’re currently subsidized about 50% so they’re cheaper and many cities had huge problems with racism making it hard to get cabs for some people or neighborhoods. At least on the latter front there are now competing apps which make that harder to do without leaving a record.
38 days ago
https://keramatzade.com/Earn-wealth-with-amazing-business-ideals https://keramatzade.com/Law-of-Attraction-of-Wealth https://keramatzade.com/Ways-to-make-money https://modirebimeh.ir/online-calculation-of-iranian-life-insurance/ https://modirebimeh.ir/engineers-professional-liability-insurance/ https://modirebimeh.ir/third-party-insurance-calculation/ https://modirebimeh.ir/iran-liability-insurance-have-you-not-yet-insured-your-business-with-iran-liability-insurance/ https://modirebimeh.ir/iran-life-insurance-ganji-for-the-future-of-children-and-families/ https://modirebimeh.ir/iran-car-body-insurance-the-best-and-most-prestigious-in-the-iranian-insurance-industry/ https://modirebimeh.ir/the-most-reliable-and-unrivaled-third-party-car-insurance-in-iran/ https://keramatzade.com/14-ways-to-increase-revenue https://keramatzade.com/8-ways-to-increase-revenue https://keramatzade.com/25-jobs-with-which-you-can-earn-up-to-a-million-dollars https://keramatzade.com/success-secret-1 https://keramatzade.com/Make-Money-Online-Effective-step-by-step-money-making-techniques https://keramatzade.com/Make-money-at-home https://keramatzade.com/Ways-to-make-money-without-capital https://keramatzade.com/Creative-Money-Making-Ideas https://keramatzade.com/The-law-of-attracting-money https://keramatzade.com/Ways-to-Make-Money-at-Home https://keramatzade.com/Immediate-absorption-of-wealth-in-10-minutes-and-attractive-ways-to-get-rich https://keramatzade.com/The-secret-of-attracting-money-in-Iran-to-achieve-creative-money-maker-ideas https://keramatzade.com/Ways-to-get-rich-in-Iran-with-the-most-wonderful-business-ideas https://keramatzade.com/Astonishing-economic-intelligence-test-to-increase-financial-intelligence
tante
251 days ago
reply
Simple list with ways Lyft and Uber are making city traffic and transportation worse.
Berlin/Germany
38 days ago
https://keramatzade.com/Earn-wealth-with-amazing-business-ideals https://keramatzade.com/Law-of-Attraction-of-Wealth https://keramatzade.com/Ways-to-make-money https://modirebimeh.ir/online-calculation-of-iranian-life-insurance/ https://modirebimeh.ir/engineers-professional-liability-insurance/ https://modirebimeh.ir/third-party-insurance-calculation/ https://modirebimeh.ir/iran-liability-insurance-have-you-not-yet-insured-your-business-with-iran-liability-insurance/ https://modirebimeh.ir/iran-life-insurance-ganji-for-the-future-of-children-and-families/ https://modirebimeh.ir/iran-car-body-insurance-the-best-and-most-prestigious-in-the-iranian-insurance-industry/ https://modirebimeh.ir/the-most-reliable-and-unrivaled-third-party-car-insurance-in-iran/ https://keramatzade.com/14-ways-to-increase-revenue https://keramatzade.com/8-ways-to-increase-revenue https://keramatzade.com/25-jobs-with-which-you-can-earn-up-to-a-million-dollars https://keramatzade.com/success-secret-1 https://keramatzade.com/Make-Money-Online-Effective-step-by-step-money-making-techniques https://keramatzade.com/Make-money-at-home https://keramatzade.com/Ways-to-make-money-without-capital https://keramatzade.com/Creative-Money-Making-Ideas https://keramatzade.com/The-law-of-attracting-money https://keramatzade.com/Ways-to-Make-Money-at-Home https://keramatzade.com/Immediate-absorption-of-wealth-in-10-minutes-and-attractive-ways-to-get-rich https://keramatzade.com/The-secret-of-attracting-money-in-Iran-to-achieve-creative-money-maker-ideas https://keramatzade.com/Ways-to-get-rich-in-Iran-with-the-most-wonderful-business-ideas https://keramatzade.com/Astonishing-economic-intelligence-test-to-increase-financial-intelligence
rocketo
252 days ago
reply
We have to stop this
seattle, wa
38 days ago
https://keramatzade.com/Earn-wealth-with-amazing-business-ideals https://keramatzade.com/Law-of-Attraction-of-Wealth https://keramatzade.com/Ways-to-make-money https://modirebimeh.ir/online-calculation-of-iranian-life-insurance/ https://modirebimeh.ir/engineers-professional-liability-insurance/ https://modirebimeh.ir/third-party-insurance-calculation/ https://modirebimeh.ir/iran-liability-insurance-have-you-not-yet-insured-your-business-with-iran-liability-insurance/ https://modirebimeh.ir/iran-life-insurance-ganji-for-the-future-of-children-and-families/ https://modirebimeh.ir/iran-car-body-insurance-the-best-and-most-prestigious-in-the-iranian-insurance-industry/ https://modirebimeh.ir/the-most-reliable-and-unrivaled-third-party-car-insurance-in-iran/ https://keramatzade.com/14-ways-to-increase-revenue https://keramatzade.com/8-ways-to-increase-revenue https://keramatzade.com/25-jobs-with-which-you-can-earn-up-to-a-million-dollars https://keramatzade.com/success-secret-1 https://keramatzade.com/Make-Money-Online-Effective-step-by-step-money-making-techniques https://keramatzade.com/Make-money-at-home https://keramatzade.com/Ways-to-make-money-without-capital https://keramatzade.com/Creative-Money-Making-Ideas https://keramatzade.com/The-law-of-attracting-money https://keramatzade.com/Ways-to-Make-Money-at-Home https://keramatzade.com/Immediate-absorption-of-wealth-in-10-minutes-and-attractive-ways-to-get-rich https://keramatzade.com/The-secret-of-attracting-money-in-Iran-to-achieve-creative-money-maker-ideas https://keramatzade.com/Ways-to-get-rich-in-Iran-with-the-most-wonderful-business-ideas https://keramatzade.com/Astonishing-economic-intelligence-test-to-increase-financial-intelligence

The Green New Deal: How We Will Pay For It Isn’t ‘A Thing’ ...

2 Shares

The Green New Deal: How We Will Pay For It Isn’t ‘A Thing’ – And Inflation Isn’t Either.

Read the whole story
graydon
264 days ago
reply
sarcozona
264 days ago
reply
Share this story
Delete

Announcing Rust 1.31 and Rust 2018

2 Shares

The Rust team is happy to announce a new version of Rust, 1.31.0, and "Rust 2018" as well. Rust is a programming language that empowers everyone to build reliable and efficient software.

If you have a previous version of Rust installed via rustup, getting Rust 1.31.0 is as easy as:

$ rustup update stable

If you don't have it already, you can get rustup from the appropriate page on our website, and check out the detailed release notes for 1.31.0 on GitHub.

What's in 1.31.0 stable

Rust 1.31 may be the most exciting release since Rust 1.0! Included in this release is the first iteration of "Rust 2018," but there's more than just that! This is going to be a long post, so here's a table of contents:

Rust 2018

We wrote about Rust 2018 first in March, and then in July. For some more background about the why of Rust 2018, please go read those posts; there's a lot to cover in the release announcement, and so we're going to focus on the what here. There's also a post on Mozilla Hacks as well!

Briefly, Rust 2018 is an opportunity to bring all of the work we've been doing over the past three years together, and create a cohesive package. This is more than just language features, it also includes

  • Tooling (IDE support, rustfmt, Clippy)
  • Documentation
  • Domain working groups work
  • A new web site

We'll be covering all of this and more in this post.

Let's create a new project with Cargo:

$ cargo new foo

Here's the contents of Cargo.toml:

[package]
name = "foo"
version = "0.1.0"
authors = ["Your Name <you@example.com>"]
edition = "2018"

[dependencies]

A new key has been added under [package]: edition. Note that it has been set to 2018. You can also set it to 2015, which is the default if the key does not exist.

By using Rust 2018, some new features are unlocked that are not allowed in Rust 2015.

It is important to note that each package can be in either 2015 or 2018 mode, and they work seamlessly together. Your 2018 project can use 2015 dependencies, and a 2015 project can use 2018 dependencies. This ensures that we don't split the ecosystem, and all of these new things are opt-in, preserving compatibility for existing code. Furthermore, when you do choose to migrate Rust 2015 code to Rust 2018, the changes can be made automatically, via cargo fix.

What kind of new features, you may ask? Well, first, features get added to Rust 2015 unless they require some sort of incompatibility with 2015's features. As such, most of the language is available everywhere. You can check out the edition guide to check each feature's minimum rustc version as well as edition requirements. However, there are a few big-ticket features we'd like to mention here: non-lexical lifetimes, and some module system improvements.

Non-lexical lifetimes

If you've been following Rust's development over the past few years, you may have heard the term "NLL" or "non-lexical lifetimes" thrown around. This is jargon, but it has a straightforward translation into simpler terms: the borrow checker has gotten smarter, and now accepts some valid code that it previously rejected. Consider this example:

fn main() {
    let mut x = 5;

    let y = &x;

    let z = &mut x;
}

In older Rust, this is a compile-time error:

error[E0502]: cannot borrow `x` as mutable because it is also borrowed as immutable
 --> src/main.rs:5:18
  |
4 |     let y = &x;
  |              - immutable borrow occurs here
5 |     let z = &mut x;
  |                  ^ mutable borrow occurs here
6 | }
  | - immutable borrow ends here

This is because lifetimes follow "lexical scope"; that is, the borrow from y is considered to be held until y goes out of scope at the end of main, even though we never use y again. This code is fine, but the borrow checker could not handle it.

Today, this code will compile just fine.

What if we did use y, like this for example:

fn main() {
    let mut x = 5;
    let y = &x;
    let z = &mut x;
    
    println!("y: {}", y);
}

Older Rust will give you this error:

error[E0502]: cannot borrow `x` as mutable because it is also borrowed as immutable
 --> src/main.rs:5:18
  |
4 |     let y = &x;
  |              - immutable borrow occurs here
5 |     let z = &mut x;
  |                  ^ mutable borrow occurs here
...
8 | }
  | - immutable borrow ends here

With Rust 2018, this error changes for the better:

error[E0502]: cannot borrow `x` as mutable because it is also borrowed as immutable
 --> src/main.rs:5:13
  |
4 |     let y = &x;
  |             -- immutable borrow occurs here
5 |     let z = &mut x;
  |             ^^^^^^ mutable borrow occurs here
6 |     
7 |     println!("y: {}", y);
  |                       - borrow later used here

Instead of pointing to where y goes out of scope, it shows you where the conflicting borrow occurs. This makes these sorts of errors far easier to debug.

In Rust 1.31, this feature is exclusive to Rust 2018. We plan to backport it to Rust 2015 at a later date.

Module system changes

The module system can be a struggle for people first learning Rust. Everyone has their own things that take time to master, of course, but there's a root cause for why it's so confusing to many: while there are simple and consistent rules defining the module system, their consequences can feel inconsistent, counterintuitive and mysterious.

As such, the 2018 edition of Rust introduces a few changes to how paths work, but they end up simplifying the module system, to make it more clear as to what is going on.

Here's a brief summary:

  • extern crate is no longer needed in almost all circumstances.
  • You can import macros with use, rather than a #[macro_use] attribute.
  • Absolute paths begin with a crate name, where the keyword crate refers to the current crate.
  • A foo.rs and foo/ subdirectory may coexist; mod.rs is no longer needed when placing submodules in a subdirectory.

These may seem like arbitrary new rules when put this way, but the mental model is now significantly simplified overall.

There's a lot of details here, so please read the edition guide for full details.

More lifetime elision rules

Let's talk about a feature that's available in both editions: we've added some additional elision rules for impl blocks and function definitions. Code like this:

impl<'a> Reader for BufReader<'a> {
    // methods go here
}

can now be written like this:

impl Reader for BufReader<'_> {
    // methods go here
}

The '_ lifetime still shows that BufReader takes a parameter, but we don't need to create a name for it anymore.

Lifetimes are still required to be defined in structs. However, we no longer require as much boilerplate as before:

// Rust 2015
struct Ref<'a, T: 'a> {
    field: &'a T
}

// Rust 2018
struct Ref<'a, T> {
    field: &'a T
}

The : 'a is inferred. You can still be explicit if you prefer. We're considering some more options for elision here in the future, but have no concrete plans yet.

const fn

There's several ways to define a function in Rust: a regular function with fn, an unsafe function with unsafe fn, an external function with extern fn. This release adds a new way to qualify a function: const fn. It looks like this:

const fn foo(x: i32) -> i32 {
    x + 1
}

A const fn can be called like a regular function, but it can also be used in any constant context. When it is, it is evaluated at compile time, rather than at run time. As an example:

const SIX: i32 = foo(5);

This will execute foo at compile time, and set SIX to 6.

const fns cannot do everything that normal fns can do; they must have deterministic output. This is important for soundness reasons. Currently, const fns can do a minimal subset of operations. Here's some examples of what you can do:

  • Arithmetic and comparison operators on integers
  • All boolean operators except for && and ||
  • Constructing arrays, structs, enums, and tuples
  • Calls to other const fns
  • Index expressions on arrays and slices
  • Field accesses on structs and tuples
  • Reading from constants (but not statics, not even taking a reference to a static)
  • & and * of references
  • Casts, except for raw pointer to integer casts

We'll be growing the abilities of const fn, but we've decided that this is enough useful stuff to start shipping the feature itself.

For full details, please see the reference.

New tools

The 2018 edition signals a new level of maturity for Rust's tools ecosystem. Cargo, Rustdoc, and Rustup have been crucial tools since 1.0; with the 2018 edition, there is a new generation of tools ready for all users: Clippy, Rustfmt, and IDE support.

Rust's linter, clippy, is now available on stable Rust. You can install it via rustup component add clippy and run it with cargo clippy. Clippy is now considered 1.0, which carries the same lint stability guarantees as rustc. New lints may be added, and lints may be modified to add more functionality, however lints may never be removed (only deprecated). This means that code that compiles under clippy will continue to compile under clippy (provided there are no lints set to error via deny), but may throw new warnings.

Rustfmt is a tool for formatting Rust code. Automatically formatting your code lets you save time and arguments by using the official Rust style. You can install with rustup component add rustfmt and use it with cargo fmt.

This release includes Rustfmt 1.0. From now on we guarantee backwards compatibility for Rustfmt: if you can format your code today, then the formatting will not change in the future (only with the default options). Backwards compatibility means that running Rustfmt on your CI is practical (use cargo fmt -- --check). Try that and 'format on save' in your editor to revolutionize your workflow.

IDE support is one of the most requested tooling features for Rust. There are now multiple, high quality options:

Work on IDE support is not finished, in particular code completion is not up to scratch in the RLS-based editors. However, if you mainly want support for types, documentation, and 'go to def', etc. then you should be happy.

If you have problems installing any of the tools with Rustup, try running rustup self update, and then try again.

Tool lints

In Rust 1.30, we stabilized "tool attributes", like #[rustfmt::skip]. In Rust 1.31, we're stabilizing something similar: "tool lints," like #[allow(clippy::bool_comparison)] These give a namespace to lints, so that it's more clear which tool they're coming from.

If you previously used Clippy's lints, you can migrate like this:

// old
#![cfg_attr(feature = "cargo-clippy", allow(bool_comparison))]

// new
#![allow(clippy::bool_comparison)]

You don't need cfg_attr anymore! You'll also get warnings that can help you update to the new style.

Documentation

Rustdoc has seen a number of improvements this year, and we also shipped a complete re-write of the "The Rust Programming Language." Additionally, you can buy a dead-tree copy from No Starch Press!

We had previously called this the "second edition" of the book, but since it's the first edition in print, that was confusing. We also want to periodically update the print edition as well. In the end, after many discussions with No Starch, we're going to be updating the book on the website with each release, and No Starch will periodically pull in our changes and print them. The book has been selling quite well so far, raising money for Black Girls Code.

You can find the new TRPL here.

Domain working groups

We announced the formation of four working groups this year:

  • Network services
  • Command-line applications
  • WebAssembly
  • Embedded devices

Each of these groups has been working very hard on a number of things to make Rust awesome in each of these domains. Some highlights:

  • Network services has been shaking out the Futures interface, and async/await on top of it. This hasn't shipped yet, but we're close!
  • The CLI working group has been working on libraries and documentation for making awesome command-line applications
  • The WebAssembly group has been shipping a ton of world-class tooling for using Rust with wasm.
  • Embedded devices has gotten ARM development working on stable Rust!

You can find out more about this work on the new website!

New Website

Last week we announced a new iteration of the web site. It's now been promoted to rust-lang.org itself!

There's still a ton of work to do, but we're proud of the year of work that it took by many people to get it shipped.

Library stabilizations

A bunch of From implementations have been added:

  • u8 now implements From<NonZeroU8>, and likewise for the other numeric types and their NonZero equivalents
  • Option<&T> implements From<&Option<T>>, and likewise for &mut

Additionally, these functions have been stabilized:

See the detailed release notes for more.

Cargo features

Cargo will now download packages in parallel using HTTP/2.

Additionally, now that extern crate is not usually required, it would be jarring to do extern crate foo as bar; to rename a crate. As such, you can do so in your Cargo.toml, like this:

[dependencies]
baz = { version = "0.1", package = "foo" }

or, the equivalent

[dependencies.baz]
version = "0.1"
package = "foo"

Now, the foo package will be able to be used via baz in your code.

See the detailed release notes for more.

Contributors to 1.31.0

At the end of release posts, we normally thank the people who contributed to this release. But for this release, more so than others, this list does not truly capture the amount of work and the number of people who have contributed. Each release is only six weeks, but this release is the culmination of three years of effort, in countless repositories, by numerous people. It's been a pleasure to work with you all, and we look forward to continuing to grow in the next three years.

Read the whole story
sarcozona
313 days ago
reply
graydon
313 days ago
reply
Share this story
Delete
Next Page of Stories