Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How get around the arithmetic overflow error converting expression to data type int?

I have the following query:

-- CTE to remove outliers, e.g. remove the fastest and slowest results
;WITH MinMaxCTE AS 
(
  SELECT ServerName, CONVERT(VARCHAR(10), UpdatedOn, 101) AS [Date], Version, 
    MIN(JaguarStartupTime) AS MinStartTime, MAX(JaguarStartupTime) AS MaxStartTime
  FROM dbo.MachineConfiguration (NOLOCK)
  WHERE DomainLogin NOT IN (SELECT DomainLogin FROM dbo.SupportGroup)
  GROUP BY ServerName, CONVERT(VARCHAR(10), UpdatedOn, 101), Version
)

SELECT  AVG(mc.JaguarStartupTime) AS AverageTime
    , COUNT(*) AS NumEntries
    , mc.Version
FROM    #Eligible mc (NOLOCK)
JOIN MinMaxCTE cte ON mc.ServerName = cte.ServerName 
  AND CONVERT(VARCHAR(10), mc.UpdatedOn, 101) = cte.[Date] 
  AND mc.Version = cte.Version 
  AND mc.JaguarStartupTime <> cte.MinStartTime 
  AND mc.JaguarStartupTime <> cte.MaxStartTime
GROUP BY mc.Version
ORDER BY Version DESC, AVG(mc.JaguarStartupTime) ASC

The definition of the #Eligible temp table is

create table #Eligible (
    Version nvarchar(50), JaguarStartupTime int, 
    ServerName nvarchar(50), UpdatedOn datetime )

No matter what condition or aggregation I comment out, I always get the following error: Arithmetic overflow error converting expression to data type int.

Where can I go from here? How do I debug this further?

EDIT: Sample data

Version JaguarStartupTime    ServerName     UpdatedOn
6.4.6.082      16040         NewOrleansLA   2012-08-08 12:34:12.330
6.5.1.012      40390         BatonRougeLA   2012-08-08 18:33:17.440
6.5.1.012      48379         HonoluluHI     2012-08-09 04:42:50.453
like image 716
AngryHacker Avatar asked Nov 02 '12 17:11

AngryHacker


People also ask

How do you fix an arithmetic overflow error int to data type numeric?

You need to increase the width of the variable to store this number e.g. making @sample NUMERIC (6,2) will solve this error.

What does arithmetic overflow error convert to data type int mean?

The error "Arithmetic overflow error converting IDENTITY to data type int" comes when the IDENTITY value is inserted into a column of data type int, but the value is out-of-range.

How do I fix an arithmetic overflow in SQL Server?

One solution is to change the data type of the IDENTITY column. For example, if it's smallint , change it to int . Or if it's already int , change it to bigint . Another possible solution would be to reset the IDENTITY seed to a lower value.

How do you solve arithmetic overflow in C++?

"Arithmetic overflow: Using operator '*' on a 4 byte value and then casting the result to a 8 byte value. Cast the value to the wider type before calling operator '*' to avoid overflow".


1 Answers

Have you tried casting your jaguarstartup times to a bigint in your avg aggregate like so...

AVG(CAST(mc.JaguarStartupTime AS BIGINT))

This should sort out the arithmetic overflow.

To calculate the mean average the server needs to be able to sum all of the ints first, so the datatype you are averaging on needs to able to store the sum of those values, even if the returned answer is within the range of of an int

like image 133
Rich Andrews Avatar answered Oct 29 '22 17:10

Rich Andrews